We are currently moving our web services and information to Canada.ca.

The Treasury Board of Canada Secretariat website will remain available until this move is complete.

Roles for Government in Evaluation Quality Assurance: Discussion Paper

Archived information

Archived information is provided for reference, research or recordkeeping purposes. It is not subject à to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

April 2006

Prepared for: Centre of Excellence for Evaluation, Treasury Board of Canada, Secretariat
Prepared by: J. Bradley Cousins, Ph. D. & Tim Aubry, Ph. D., Centre for Research on Community Services, 
University of Ottawa

Contact:
B. Cousins, 145 Jean Jacques Lussier, Faculty of Education, University of Ottawa, K1N 6N5, bcousins@uottawa.ca
(613) 562-5800 ext. 4036, fax (613) 562-5146. 

The authors express their sincere gratitude for assistance in the research of Foong-Peng Leong.

© Her Majesty the Queen in Right of Canada, represented by the President of the Treasury Board, 2006, and the University of Ottawa 
Catalogue No. BT22-108/2006
ISBN 0-662-49282-X

This document is available in alternative formats and on the Treasury Board of Canada Secretariat's Web site at the following address: http://www.tbs-sct.gc.ca/cee/

The online version contains detailed information for each of the university or government evaluation training programs available outside of Canada and within as described in Table 2 and 3 (see Appendix A online) and for each of the centres of excellence in evaluation and related fields available outside of Canada and within as described in Table 5 and 6 (see Appendix B online).


Executive summary

1. Introduction
        1.1 Context
        1.2 Rationale
        1.3 Overview

d 2. Review and Integration of Literature on Professionalization of Evaluation
        2.1 Evaluation and Government
        2.2 Professionalization Debates
        2.3 Core Competencies for Evaluators
        2.4 Implications for Professionalization of Evaluation in Canada

3. Survey of University-based Evaluation Training Programs and Options
        3.1 Evaluation Training beyond Canadian Borders
        3.2 State of the Art of Evaluation Training in Canada
        3.3 Case Profile of a Government/University Collaborative Program

4. Survey of University-based Evaluation Centres of Excellence
        4.1 University-based Centres of Excellence with an Interest in Evaluation located beyond Canadian Borders
        4.2 Canadian University-based Centres of Excellence with an Interest in Evaluation

5. Conclusions and Implications
        5.1 Training and Education
        5.2 Centres of Excellence
        5.3 Other Involvement with Universities
        5.4 Support for Professional Society
        5.5 Closing Remark

References

Appendix A

Appendix B


Executive summary

Overview

In December 2005, the Treasury Board Secretariat's (TBS) Centre of Excellence for Evaluation (CEE) commissioned the services of the Centre for Research on Community Services (CRCS) at the University of Ottawa to develop a discussion paper that identifies for consideration actionable strategies and options for government to foster advanced professional development for evaluators in the interests of enhancing quality assurance in the evaluation function. Of particular interest are implications for government's relationship with the university sector.

To accomplish these objectives, the paper is divided into four sections. In the initial section of the paper, a review of the literature on professionalization of evaluation is conducted. The second section of the paper presents results of a survey of university-based evaluation training programs and options based on internet searches and telephone consultations. The third section of the paper provides results of a survey of existing university-based centres of excellences in Canada and internationally based on internet research, bibliographic follow-up, and telephone and e-mail consultations. Based on the first three sections of the paper, the final section of the paper provides a series of options for CEE to build evaluation capacity through partnerships with universities.

 

Review and Integration of Literature on Professionalization of Evaluation

A review of the literature on the role of evaluation in government found evaluation to be recognized as an important and longstanding function in Canadian government. However, the extent that evaluation has actually integrated into government decision-making has been limited. The current management framework (i.e., TBS Management Accountability Framework) adopted by the Canadian federal government provides an opportunity for evaluation to become a core function of public management.

The extent that the Canadian federal government has influenced the development of the field of program evaluation and evaluation capacity in Canada is less clear and direct than in the United States and United Kingdom. The establishment of the Canadian Evaluation Society in 1981 and, subsequently, the publication of Canada's bilingual peer-reviewed outlet the Canadian Journal of Program Evaluation were very much tied to federal evaluation concerns. The establishment of the CEE in 2001 represents the Canadian federal government's primary contribution to evaluation capacity building.

A review of the literature on the professionalization of program evaluation suggested that program evaluation cannot be yet considered a bona fide profession because it lacks certification or licensure processes, criteria for determining membership to professional associations, and pre-service training programs of evaluators that are recognized and formally accredited by professional associations of program evaluation.

Identified obstacles or impediments for program evaluation to become a bona fide profession include the increased involvement of non-evaluators in evaluation activities, the lack of clarity around the definition of evaluation, the diverse and un-patterned career path of program evaluators, the ambivalence with adopting certification among evaluators, the paucity of university-level training programs, the costs of implementing a certification system, the problems associated with certifying or "grandparenting" current program evaluation practitioners based on their training and experience, and the increased risk of litigation against program evaluators encouraged by certification.

The field of program evaluation does not appear ready for individual-level certification in the form of licensure or certification by the professional society. Based on work by a task force of the American Evaluation Association, a credentialing system seems more feasible at this point in time and can serve as a transitory or intermediary step toward a more stringent certification system. In this system, individuals receive credentials for completing a set of courses or experiences, or combination thereof.

A review of the literature on core competencies for evaluators found two recent research projects identifying empirically a set of core competencies. The first has emerged from a group of American researchers led by King and Stevahn (King, Stevahn, Ghere & Minnema, 2001; Stevahn et al., 2005a, 2005b) and has resulted in an empirically validated set of 'Essential Competencies for Program Evaluators' (ECPE) that evolved over a five-year period and was revised on the basis of input from American and Canadian evaluators and experts in the field. These competencies are organized under six categories or themes: (1) professional practice (6 competencies), (2) systematic inquiry (20 competencies), (3) situational analysis (12 competencies), (4) project management (12 competencies), (5) reflective practice (5 competencies), (6) interpersonal competence (6 competencies).

A second set of competencies was derived from another empirically grounded inquiry commissioned by the Canadian Evaluation Society in support of its evaluation advocacy agenda. Known as the Core Body of Knowledge (CBK) project and undertaken by a group of researches led by Zorzi (Zorzi, Perrin, McGuire, Long, & Lee, 2002; Zorzi, McGuire & Perrin, 2002; see also McGuire & Zorzi), the project produced a list of 23 general knowledge and skill elements of program evaluation, within which more specific knowledge, skills and practices were identified. Each element was categorized into one of the following clusters: ethics, evaluation planning and design, data collection, data analysis and interpretation, communication and interpersonal skills, and project management.

Based on the state of the research on the professionalization of program evaluation, it is concluded that there is a clear and important role for the federal government to play in developing the professionalization of evaluation and enhancing quality assurance in evaluation within government and beyond. It would seem prudent to continue with efforts within the professional development approach towards certification that would capitalize on the fine work on the development of evaluator competencies that has been completed to date. Implicated would be government support for the development of a credentialing system that would provide some basis for deciding whether those responsible for carrying out evaluations have sufficient background training and experiences to conduct evaluations that meet a high standard of quality and effectiveness.

 

Survey of University-based Evaluation Training Programs and Options

A survey of training programs abroad revealed that most identifiable university-based evaluation programs are located in American Universities (exceptions in Melbourne and London) and that these may be somewhat on the decline, due perhaps to the point in the career trajectory of the founding members of the programs. Nevertheless, several graduate degree and certificate programs in evaluation were located. There exist many options for ongoing professional development and continuing education in evaluation but most of these do not lead to formal certification of achievement, as opposed to participation. Exceptions would be the Training Institute run by the US GAO and single evaluation graduate courses offered in many university departments and faculties.

Evaluation training opportunities in Canada are widely available but opportunities for advanced level university training appear to be quite limited. There currently exist no degree programs in evaluation in the country and only three graduate certificate programs (one is pending approval) and one diploma program at a community college. While a wide array of universities offer graduate study in evaluation, this is most often limited to course-level experiences. Such courses may be integrated into degree programs (concurrently or subsequently) and it is likely that candidates could specialize in evaluation in degree programs in related disciplines such as education or applied social psychology. It is encouraging to note, however, that university courses in evaluation and related topics exists on such a broad basis and that several universities offer more than one evaluation course within single faculties or departments. The potential for certificate program development, for example, would be increased in circumstances where faculties or departments could build on existing courses rather than developing programs from scratch. Finally, a wide variety of other training and professional development opportunities in evaluation exist both inside government and out, but at present there are no regulations requiring candidates to have undergone such training in order to hold evaluation-related posts within the federal government.

An M.Sc. program in the UK that represents a partnership between the federal government and a university was described as a case profile. This program is quite unique and bears quite directly on considerations of government's role in fostering evaluation quality assurance. As partners, both the government and the university are implicated in program development and delivery responsibilities. While the 2 years part-time program appears to be highly relevant to the needs of the government, it also meets university sector standards for the graduate degree.

 

Survey of University-based Centres of Excellence with significant interests in Evaluation

A wide range of centres in five English speaking countries around the globe were located. Many of the centres were located in faculties of education or human development but, health services and interdisciplinary centers were also noted to have a presence. Centres varied quite substantially in size and in the scope of their work. Most were involved in some combination of research, evaluation related practices including consultation, service delivery and dissemination, and training or education. Most centres were dependent in some way shape or form on government (usually federal or state) for sponsorship, source of competitive grant funds, or contracted project work. Private foundations often provided support as well. Centre business often included disciplinary research (e.g., child welfare, public health) in addition to evaluation-related services. In some instances formal links to degree programs but sometimes center activities did not involve education or training.

Compared to the international sample, Canadian centres of excellence with significant interest in evaluation-related activities, appear to be somewhat more homogeneous in size and less prevalent in faculties of education. We observed a tendency for interdisciplinary centers to exists, in which the centres do not appear to be affiliated with a particular disciplinary university faculty or department. Centres that participate in evaluation-related activities were difficult to locate by virtue of evaluation not being represented in the centre name. Nevertheless, there is substantial involvement of university-based centres in Canada in evaluation activities, either in consultation, service delivery, or training. There is also a good deal of interest in fostering evidence-based practice in the respective field of practice, in some cases through disseminating policy research or brokering research done elsewhere. Finally, it seems clear that centres are dependent to a significant degree on funds generated through their relationship with government, either as a recipient of sponsorship, grant recipient or as a contractor to government at provincial and federal levels.

 

Conclusions and Implications

Implications of the findings in the discussion paper for training and education include developing pilot projects in universities of graduate certificate programs in program evaluation. Support in the form of guaranteeing a certain number of federal government placements (i.e., government personnel to be retrained for evaluation) over coming years would be useful to help establish and develop the programs within the university structure. Such a project might lead to a defacto credentialing system for government to use for hiring and contracting purposes. Other training and education options include the federal government partnering with universities to develop degree programs as well as having CEE continuing to offer workshops and short courses and continuing to support CES Essential Skills Series.

The development of centres of excellence in Program Evaluation located at Canadian universities is also suggested as a vehicle for developing evaluation capacity. These would serve to involve more academics in program evaluation and could foster the development of highly qualified personnel in evaluation through practical and research experiences for post-doctoral candidates and graduate students. In addition, the continuation and expansion of liaison between academics and the federal government is recommended through their participation in quality assurance activities such as advisory committees and peer-review functions related to program evaluation in the federal government.

Other suggested involvement with universities includes developing exchanges between government and academe such as through secondments, sabbatical placements, and short-term leave replacements and by supporting student development by providing work placements and internship sites.

Finally, the support of the federal government of CES in developing a credentialing system is worth consideration. This can be viewed as an incremental strategy toward eventual professional certification.

 

1. Introduction

1.1 Context

In December 2005, The Treasury Board Secretariat's Centre of Excellence for Evaluation (CEE) commissioned the services of the Centre for Research on Community Services (CRCS) at the University of Ottawa to develop a discussion paper that provides options for moving forward on evaluation credentialing and the establishment of Centres of Expertise in Evaluation within universities.

In the Statement of Work, CEE set the following three objectives for the discussion paper: (1) To provide a global perspective of the state of the art of university level evaluation training and credentialing by identifying the type of training and accreditation available in various jurisdictions, (2) to relate the findings of the global perspective to the Canadian context by identifying what relevant training is currently available in Canada (university based, Canadian Evaluation Society course offerings, federal government course offerings), and (3) to provide different options for consideration by CEE that would allow a cost-effective way to build evaluation capacity within the federal government through partnerships with universities.

 

1.2 Rationale

A recent study by Gussman (2005), commissioned by the CEE, examined improving the professionalism of evaluation in the federal government. Gussman considered developments and trends in program evaluation against the context of evolving public service management and provided options for improving the credibility of evaluation practitioners. Such issues, which centre on enhancing quality assurance in evaluation are becoming increasingly important in the federal government for a variety of reasons.

First, the new Federal Accountability Act and Action Plan , introduced on April 11, 2006, is intended to make government more effective and accountable by bringing forward specific measures to help strengthen accountability and increase transparency and oversight in government operations. This remains consistent with the current TBS Management Accountability Framework (MAF) in which expectations for modern public service management are defined. The underlying management framework Results for Canadians (Government of Canada, 2000) requires facility with results-based management (RBM) which implicates the use of evaluative inquiry to agree on, measure, and report on results (Auditor General of Canada, 1997). While there have been prior debates about the relationship between performance measurement and evaluation, Segsworth (2005) makes the case that Treasury Board requirements for Results-based Management and Accountability Frameworks (RMAF) and Program Activity Architectures (PAA) recognize and promote a much closer relationship between the two functions. RMAFs are required to accompany Treasury Board Submissions of grant and contribution programs to assist in monitoring, evaluating and reporting on program results. PAAs reflect how a department allocates and manages resources under its control to achieve intended results and how programs and related activities are integrated with the department's strategic outcomes. If the evaluation function is to become truly integrated into the broader management function in government, it will be necessary to ensure the function is recognized to have a unique contribution to be made (apart from that of internal audit, for example) and that quality evaluation is assured.

Second, according to Aucoin (2005),

The quality of program evaluations is due to the quality of the staff who carry out this function, the resources devoted to it, and the extent to which the functional community is developed and maintained as a professional public service community. 

(p. 21)

Yet recent evidence from another study commissioned by the CEE suggests that evaluation is not well integrated with senior management decision-making. Breen and associates (2005) interviewed Deputy Ministers and found that although they consistently expressed the view that evaluation is a policy/program function, they identify a lack of a feedback loop between evaluation findings and policy/program development and management. Deputies are also aware that evaluation capacity was hit hard during the program review exercise of the mid 1990s and that capacity is limited today. They commented that most evaluation studies are contracted out due a lack of internal resources and, as such, this weakens the ability of evaluation units to become ongoing sources of advice and wisdom. Aucoin (2005) maintains that the enhancement of quality in program evaluation will not happen unless there is the demand for quality from senior officials or ministers.

Third, despite Treasury Board of Canada Secretariat (TBS) initiatives intended to improve evaluation quality, such as the establishment of the CEE in 2001, evidence has accumulated to show that evaluation quality is limited in several respects. An internal study of the quality of evaluations across departments and agencies carried out by CEE (2004) showed some improvement in evaluation reporting since 2002, suggesting that TBS's efforts to improve the quality of evaluations are meeting with some success. Yet significant weaknesses in evaluation reporting continue to be observed. These included: neglect in specifying the evaluation issues being addressed; superficial coverage of cost-effectiveness issues; lacking descriptions of methods used; lack or absence of integration of data from performance measurement systems; and the like (CEE, 2004). According to Segsworth (2005) concerns about quality have been raised by the Auditor General in just about every audit of the evaluation function.

Fourth, Gussman (2005) notes that the internal audit function within the Canadian federal government is currently being strengthened in the wake of various inquiries and that this in turn may signal to the evaluation community the need to take overt steps toward quality assurance by way of raising the profile of the evaluation function. Moreover, the new Policy on Internal Audit, which came into effect April 1, 2006, provides guidelines for expected qualifications for Chief Audit Executives, they are to have a Certified Internal Auditor or a professional accounting designation (CA, CGA, CMA), and for internal auditors.

…internal auditors [are to] have appropriate professional qualifications and skills, and opportunities for sufficient training and development to maintain and develop their internal auditing competence and to obtain Certified Internal Auditor certification.

(Treasury Board of Canada, Secretariat, 2006 b, Section 3.1.9)

Finally, of some 284 FTE's in evaluation throughout federal departments and agencies, many (perhaps 20%) will be retiring within the next few years and there is a pressing need if not opportunity to rejuvenate the workforce with personnel qualified and trained in evaluation. There is also increased demand for qualified personal due to ongoing staff turnover.

Given these circumstances, Gussman (2005) supported the concept of university-based programs offering a certificate in evaluation as a potential solution to improving consistency and methodological rigor within the evaluation function. The CEE wishes to build on the Gussman paper by identifying and considering actionable strategies and options for government to foster advanced professional development for evaluators in the interests of enhancing quality assurance in the evaluation function. This represents the central purpose of the present paper. Of particular interest will be implications for government's relationship with the university sector and the professional society, the Canadian Evaluation Society (CES). 

 

1.3 Overview

There has been considerable debate and discussion in the evaluation literature about the role of evaluation in government and issues concerning the professionaliztion of the field including the prospect of certification of individuals and accreditation of evaluation training programs. Much of this work arises from deliberations among colleagues with the American Evaluation Association (AEA), although the issues have most certainly captured the attention of the CES and continue to do so. We begin with a review and integration of this literature.

Next we provide a global perspective of the state of the art university-level evaluation training and credentialing. Specifically the discussion paper will identify the type of training and accreditation available in various jurisdictions beyond Canadian borders. We then report on the state of the art of training programs in Canada relating the findings of the global perspective to the Canadian context. Specially, the discussion paper will identify what relevant training is currently available in Canada (university based, Canadian Evaluation Society course offerings, federal government course offerings) and provide an analysis, based on the findings from the global perspective, of what could be developed in the Canadian context. Of particular interest is the involvement of government in evaluation training initiatives. To that end, a specific masters program in the United Kingdom representing a partnership between Cabinet Office and the Institute of Education at the University of London, will be profiled for consideration.

Evaluation training options that will be adequate to the emerging needs of the federal government represent a significant focus for potential government involvement, but there are other possible considerations that would be consistent with an agenda of enhancing quality assurance in evaluation. Specifically, university-based centres of excellence in evaluation will be another major focus of our attention in the current paper. How can such centres help to improve quality assurance in evaluation? What are some possible roles for government in promoting quality assurance through this means? In the third section we address these questions by exploring the development, mission, sustainability and function of university-based centers of excellence in evaluation beyond Canadian borders as well as within Canada.

In the final section of the paper we provide options for consideration that would allow a cost-effective way to build evaluation capacity within the federal government, such as addressing its internal capacity needs through partnerships with universities. We consider how the proposed options could be implemented by the CEE.

 

2. Review and Integration of Literature on Professionalization of Evaluation

We address three main themes in this literature review. The first is the role of evaluation within the context of government including mutual influences between evaluation and government. Next, we summarize professionalization debates in evaluation with the goal of identifying the current state of thinking in the field. Such debates have taken place predominantly within the evaluation community at large, although they are of high interest to government. Finally, we focus more narrowly on one issue that has been a central consideration in professionalization debates, namely progress toward the development of an accepted set of competencies for evaluators. Most of the published work that we located on each of these themes originates in Canada and the United States although we note increasing interest in related topics beyond North America (Russon & Russon, 2005; Perrin 2005). Documents for the literature review (discussion papers, published articles and chapters) were located primarily through consultation and bibliographic follow up.

2.1 Evaluation and Government

Perhaps the most comprehensive examination of the relationship between evaluation and government was compiled by Datta (2003). This chapter, which restricts its scope to evaluation and government in the US, was characterized by Datta as a work in progress, but we agree with her that it represents one of the few serious analyses of the ways evaluation and government have influenced one another.

The author elaborated eight ways in which government influences evaluation, several of which are particularly relevant to the current discussion. The eight identified influences are: (1) demand for internal evaluation within government; (2) demand for internal evaluation for recipients of federal funds; (3) demand for external evaluation for recipients of federal funds and requirements for impact evaluation; (4) influences on evaluation methods, designs and measures; (5) development of evaluation as a profession; (6) employment opportunities for evaluators; (7) leadership in evaluation; and (8) influences on evaluation capacity. Most of the categories of influence listed resonate well within the Canadian evaluation community. Government has provided considerable impetus for the development of the field, including the creation of a demand for evaluation, promotion of evaluation as a professional entity and the development of evaluation capacity (see, e.g., Segsworth, 2005 and Cousins' 2005 interview with J. Hudson).

It is beyond the scope of this paper to explore in any detail the history of evaluation within government in Canada. Interested readers might consult two important papers on this topic by Muller-Clemm and Barnes (1997) and, more recently, Segsworth (2005). Another excellent piece by Aucoin (2005) - a discussion paper commissioned by the CEE - situates evaluation within the context of changing structures and approaches to federal-level decision making and resource allocation.

In Canada, similar to the case in the US as portrayed by Datta (2003), government has played a key role in creating the demand for evaluation by requiring that money be set aside for evaluation of grant and contribution programs and more recently, by developing a federal management accountability framework, within which evaluation is situated. Aucoin (2005, pp. 10-11) says it well: 

With program evaluation an integral part of results-based management, the focus of program evaluation is on the management of program performance as well as on program effectiveness. The program evaluation function serves the management process alongside other current initiatives (the [Modern Comptrollership Initiative], the Management Accountability Framework, the [Strengthening Public Sector Management initiative], and Program Activity Architecture) that are part of the results-based management regime. With the advent of Expenditure Review as a continuing process, however, program evaluation can and should be a major element in government decision-making. The development of Expenditure Review offers the opportunity to establish a budgeting system that, as an integral part of the government's decision-making process, allocates resources in part on the basis of evidence on program effectiveness.

Recently, the federal government, under the leadership of Prime Minister Stephen Harper, introduced the Federal Accountability Act and Action Plan on April 11, 2006 (Government of Canada, 2006). This legislation is intended to strengthen the current system of oversight and management and will involve the review of all federal grants and contributions.

It will require that every department review, at least once every five years, the relevance and effectiveness of each ongoing grants and contributions program for which it is responsible. Grants or contributions to individuals, corporations, and non-government organizations account for $26 billion in annual transfer-payment spending. The Treasury Board will determine the scope of these reviews, how they are approached, and when departments will submit reports to the Treasury Board of Canada Secretariat.

(Government of Canada, 2006, p. 28)

Yet, despite overt internal calls for evaluation, there has been an ebb and flow in the extent to which evaluation has been integrated into government decision-making over the years. Aucoin argues that evaluation's primary virtue is its potential to generate evidence in support of program effectiveness and that the focus on this criterion in government decision making has waxed and waned over the years. In addition to public responsiveness and fiscal discipline, program effectiveness represents a third criterion for good governance and public management, one on which Canada has not fared as well as it has on the other two, in part because of changes in management and decision making policies and priorities over the years. Nevertheless the current structure provides considerable impetus, and indeed opportunity, for evaluation to be integrated as a core function of public management.

The extent to which government policy and practice has shaped evaluation methods, designs and measures is not as clear. In the US, Datta argues, there has been an evolution of methods over time, ranging from randomized control trials (RCT), quasi experimental methods, theory-based designs, multi-site evaluations and the integration of quantitative and qualitative methodologies. Federal requirements for evaluation have played some role in shaping such approaches. It is interesting to note that since Datta's (2003) chapter was published, there has been a shift in US federal policy that has created considerable turmoil in the American evaluation community.

Specifically, the US Department of Education recently decided to privilege evaluation proposals based on RCT designs (2003) and require strong justifications for the use of alternative comparative group designs. This decision has touched off enormous debate and controversy in the evaluation community (e.g., Scriven, 2003), and prompted an official response from the American Evaluation Association (2003). Regardless, it remains a clear illustration of government's influence on evaluation. Similar observations can be made in the United Kingdom. According to Gussman (2005), with its commitment to evidence-based policy making, the government of the UK has shown leadership through the use of advanced methodological approaches to policy evaluation. 

Beyond the traditional tools employed in conducting impact evaluations to assess outcomes… research designs now include: randomized control trials; regression discontinuity designs; single group pre-and-post test designs; interrupted time series designs; and regulatory impact assessments. Of these approaches, only the latter design has been commonly employed in the Canadian context.

(Gussman, 2005, p. 10)

The UK, according to Gussman, has also relied on qualitative evaluation methods and integrated mixed-method approaches into implementation evaluation designs. A similar pattern has been observed in Canada with regard to implementation evaluation, yet the impact of federal policy on outcome or impact evaluation may not yet have materialized. The accountability frameworks currently in place encourage the integration of evaluation with performance measurement and privileges outcome or impact evaluation over other approaches that are geared to understanding program implementation with the goal of improving performance and results. However, Segsworth is of the view that evaluation has yet to commit to impact evaluation in a manner consistent with the tenets of RBM:

… evidence suggests that rather little has changed in terms of the issues addressed by evaluators over time. Despite a policy mandate to deal with results, most evaluation studies continue to emphasize operational concerns.

(2005, p. 184)

Government in the US played a very significant role in the development of evaluation as a profession, suggests Datta (2003). Influence of this sort took different forms, including key players working in government, sponsorship of early conferences on evaluation, and indirectly, providing the impetus for interested parties to meet, ultimately leading to the establishment of the Evaluation Research Society, a forerunner to the current AEA. Such influences have been observed in Canada as well. The establishment of the Canadian Evaluation Society in 1981 and, subsequently, the publication of Canada's bilingual peer-reviewed outlet the Canadian Journal of Program Evaluation were very much tied to federal evaluation concerns (Cousins 2005 interview with J. Hudson).

Canadian government involvement in developing evaluation capacity has been somewhat less clear than it has in the US. South of the border, according to Datta (2003), the National Science Foundation has been directly involved in encouraging training for evaluators in math and science through support for university degree programs (J. Altschuld, personal communication, April 3, 2006). They have also provided fairly direct support to professional development opportunities such as the Evaluator's Institute and annual NSF-sponsored summer institute at Western Michigan University. Western Michigan University is also the home of the Evaluation Center, one of two - the other being the Center for the Study of Evaluation at the University of California, Los Angeles - that have received fairly extensive and consistent government support in part to build and sustain evaluation capacity for many long years (A. Gullickson, personal communication, April 11, 2006).

The Joint Committee for Educational Evaluation Standards, an organization for which federal support has been direct and substantial, is centered at the Evaluation Center and credited with the development of the widely used Program Evaluation Standards. Finally, the US GAO established in 1988 its own Training Institute to separate training and education from career counselling and organizational support in order to better meet the information needs of Congress. According to Kingsbury and Hedrick (1994), in order to do the GAOs audit and evaluation work, every evaluator must obtain a minimum of 80 hours of training every two years. Due to shifts in resource priorities within the federal government, eventually the Training Institute was dissolved but was later reconstituted as the Center for Learning. Evaluation training continues to be offered under the Center for Learning which is subject to Government Audit Standards, including the requirement of 80 hours of instruction every two years (N. Kingsbury, personal communication, April 7, 2006).

In Canada, the Canadian School of Public Service provides internal training nation-wide for Canadian federal employees, and periodically offers courses on evaluation. The establishment of the CEE in 2001 represents the Canadian federal government's primary contribution to evaluation capacity building. In addition to government influences on evaluation, evaluation has also been observed to influence government in both direct and indirect ways. Such influences are discussed by Datta, with reference to the situation in the US. Academics have been involved in an advisory capacity to evaluation studied conducted by the government in the US for a long time. The advice of such experts is very directly reflected in RFPs for evaluations and other documents.

In Canada, in recent years, academics have become members of departmental or agency evaluation advisory committees and have participated as peer reviewers of evaluation frameworks as well as of final evaluation reports. Other influences suggested by Datta are less obvious in Canada. According to her, "leaders in the academic evaluation community have enormous influence on the federal government through the students they train who later accept federal employment" (2003, p. 357). In addition, some evaluation academics in the US have been observed to have accepted roles in government, undoubtedly another avenue for the infusion of academic perspectives and knowledge as influence on the nature and consequences of evaluation in government. In Canada, these indirect influences are likely to be more subtle and somewhat muted by virtue of the comparative paucity of dedicated university-based evaluation training programs in the country and the small number of academics involved in program evaluation research. Yet, similar to many other jurisdictions, national and regional conferences sponsored by CES have provided a means for vibrant exchange about evaluation-related topics and interests between the evaluation community at large and government.

Even less certain has been the influence of evaluation on government in terms of the use of the results of studies and impact of research. The domain of inquiry of 'evaluation utilization' has been a comparatively vibrant focus for inquiry over a very period of time. Early national studies in the US (e.g., Alkin, Kosecoff, Fitz-Gibbon, & Seligman, 1974; Patton et al., 1977; Weiss & Bucuvlas, 1980) helped shape understanding about the problem of non-use of evaluation, factors and conditions supporting use and variation in the types of use of evaluation. Although direct, large scale empirical inquiry appears to have tapered off somewhat, recent publications suggest that government utilization of evaluation remains a focus of interest and debate (e.g., Grasso, 2003; Leviton, 2003). In Canada, direct inquiry into the impact of evaluation has been somewhat more limited, although there may be a growing interest in the use of social sciences research evidence in government (e.g., Amara, Ouimet, & Landry, 2004; Landry, Amara & Lamari, 2001).

The recent internal study mentioned above and commissioned by CEE (Breen & associates, 2005) provides an exception. This study focused on the views and opinions of Deputy Ministers about the evaluation function and its impact on decision-making in the federal government. Deputy Ministers provided their perceptions about the current state of the evaluation function and forces that affect it, including a general concern about a lack of a "feedback loop between evaluation findings and policy/program development and management" (Breen & associates, 2005, p. 5). While evaluators had developed useful recommendations, there did not appear to be an established way to incorporate these recommendations into policies and programs. Another ongoing study has as its focus understanding the extent to which evaluation is integrated into the organizational culture of government and how that might be improved. Cousins et al. (2006) conducted a concept mapping study to develop understanding about the conditions under which evaluation is likely to be useful and to compare the perspectives of senior program decision-makers and evaluation heads from across government departments and agencies. The study found that the two groups held remarkably similar views and that the top- ranked conditions under which evaluation is likely to be useful are: (1) conducted evaluations are of high quality / credibility / integrity; (2) evaluation has clear support for decision making and action; (3) organizational infrastructure and resources for evaluation activities are adequate; and (4) evaluation is owned, understood and embraced by users. These results are intended to feed into a second stage of research involving a planned ongoing questionnaire survey of evaluators who participated in training, and multiple case studies of government and not-for-profit sector organizations focusing on conditions that facilitate the use of program evaluation.

To summarize, evaluation has been recognized as an important function in government for a long time, probably well before the first evaluation policy in Canada, which was established in 1977. Yet the extent to which evaluation to date has been integrated as a core function into government decision making is limited. The current management framework adopted in the federal government and the new Federal Accountability Act and Action Plan provide important opportunities for making progress towards this end. The relationship between evaluation and government is dynamic and influence is likely to be reciprocal. Such relationships have been comparatively under-researched in Canada, relative to other jurisdictions.

 

2.2 Professionalization Debates

Is evaluation a profession? Is it becoming one? Should it become a profession? What would it take to affirm that it has achieved professional status? These questions, and many similar ones, have been raised on an almost cyclic basis for at least the past quarter century and they continue to capture the attention of members of the evaluation community and other interested parties (e.g., recent debates in 2005 hosted by the CES National Capital Chapter in Ottawa and the Joint Meeting of CES and AEA in Toronto). By most counts, despite excellent work in the development of professional standards and principles of practice in North America (AEA Guiding Principles for Evaluators . Shadish, Newman, Schierer & Wye, 1995; Guidelines for Ethical Conduct , CES n.d. b; Program Evaluation Standards , Joint Committee, 1994), evaluation cannot yet be thought of as a bona fide profession. (Love, 1995; Smith, 2003; Worthen, 2003).

Worthen (2003) lays out a set of criteria for a field of activity like program evaluation to be considered a profession. An adaptation of this list appears in Table 1 in the form of questions. Alongside these questions are Worthen's conclusions, an assessment with which we are inclined to agree. To summarize, the field of program evaluation boasts several characteristics of a bona fide profession, yet it is lacking on three essential criteria. On the affirmative, there is little dispute that evaluation as a domain of practice responds to an identified need and that there exists a corpus of knowledge and skill that is unique to this specialization. The emergence of several peer-reviewed journals in the field and the recent publication of the Encyclopaedia of Evaluation (Mathison, 2005) are testament to that. Formalized preparation programs exist, many of them in the US but some in other jurisdictions including Canada (see section 3.0 below). There are stable career opportunities in the field, both internally in government and other public and private sector organizations and in the consulting milieu, and the function can be considered to be institutionalized in many respects, the Canadian federal government being a primary example. There exist longstanding professional associations in Canada, the US and abroad, with more developing each year. Presently, the umbrella organization, the International Organization for Cooperation in Evaluation lists, member country-specific (e.g., CES, AEA) or regional (e.g., African Evaluation Association, Australasian Evaluation Society, European Evaluation Society) member organizations, and several of these societies have developed standards or principles of professional practice

Table 1: Criteria for Deciding if Evaluation is a Profession
(adapted from Worthen, 2003)

Criteria Current status
1. Need for evaluation specialists? Yes
2. Content (knowledge and skill) unique to evaluation? Yes
3. Formal preparation programs? Yes
4. Stable career opportunities in evaluation? Yes
5. Institutionalization of the function of evaluation? Yes
6. Certification or licensure? NO
7. Appropriate professional association? Yes
8. Exclusion of unqualified persons from membership in professional association? NO
9. Influence of evaluation association on pre-service preparation of evaluators? NO
10. Standards of practice of evaluation? Yes

Yet, nowhere does there currently exist association-level certification or licensure processes in evaluation. None of the aforementioned professional associations restricts membership on the basis of professional qualifications. Nowhere are pre-service or in-service training programs formally accredited by professional evaluation organizations. These final three features delineate how evaluation parts company with other professions and professional bodies, some - such as Canadian Association of Management Consultants (CAMC) and Canadian Institute of Chartered Accountants (CICA) - which share professional interests and turf with evaluation (see the excellent comparative paper by Long & Kishchuk, 1997). And they represent the main issues on which much of the professionalization debate has taken place.

Most of these issues have surfaced and been debated at length over the years. The AEA can be credited with considerable leadership in this area. A special issue of New Directions in Evaluation on the preparation of evaluators was published in 1994 (Altschuld & Engle, 1994). A variety of topics including prospects for certification, professional ethics, evaluation knowledge and skill sets and training programs was discussed quite directly in that issue. Later in the 1990s under the leadership of Association president Bickman, a task force, led by Altschuld was struck to explore the issue of certification more directly and deeply. A thematic section of the American Journal of Evaluation (1999, volume 20, see especially Alschuld, 1999) was devoted to the product of the work of that task force. Many of the issues were re-visited in subsequent publications such as the International Handbook on Educational Evaluation (Kelleghan, Stufflebeam & Wingate, 2003). So, given all of the discussion, what are the primary impediments or obstacles that evaluation would need to clear in order to become a bona fide profession and why has this not happened to date? To follow is a summary of some of the main challenges and issues that need to be confronted:

  • Diversification of the field - Evaluation as a domain of practice and inquiry has evolved quite remarkably over the years from its early close adherence to traditionalistic social sciences research models. Part of the explanation for this is associated with bourgeoning of epistemological challenges in the social sciences more generally. The advent of collaborative, participatory and empowerment approaches to evaluation represent a key feature of this diversification. Practical, political and philosophical justifications for involving non-evaluator program stakeholders (e.g., managers, implementers, intended program beneficiaries) have served as drivers for such change (Smith, 2003; Worthen, 2003). Perrin (2005) argues that evaluation is a transdiscipline and that it is not necessary that individual evaluators be highly knowledgeable about all areas.
  • Lack of clarity about what is evaluation - Given the emergence of such diversity, standard definitions of evaluation such as 'systematic inquiry for the purpose of judging program merit, worth and significance and to support program decision making' are no longer all-encompassing. Evaluation purposes now extend well beyond organizational, program and policy problem solving to include transformative agendas with interests in, for example, of the amelioration of social inequity and the improvement of social conditions. The lines between evaluation and program development have become increasingly obscured (Morris, 2003; Perrin, 2005)
  • Un-patterned career path - A set career path for evaluation does not exist. Many practicing evaluators and evaluation scholars entered the field through incidental, fortuitous or otherwise ad hoc means. Many have learned or continue to learn to do evaluation on the job with no particular training in the area, apart from, for example, some background in social science research methods (Borys, Gauthier, Kischuk & Roy, 2005). Many members of professional evaluation associations are not full-time evaluators. Other responsibilities might include program management, service provision, teaching or academic pursuits (Borys et al., 2005; Smith, 2003; Worthen, 2003).
  • Paucity of university-level training programs - Over the years, there have been reports of the existence of as many as 40 university-based training programs on program evaluation in the US and beyond (Altschuld, Engle, Cullen, Kim, & Macce, 1994; May, Fleisher, Schreier, & Cox, 1986). There is some recent evidence to show that evaluation university-level training programs may be on the decline in the US (Engle & Alschuld, 2003; in press). Regardless, the numbers of available programs is generally thought to be too low to support an evaluation training and certification process (Altschuld, 2005; Worthen, 1999, 2003). In Canadian universities, evaluation is taught at the level of (mostly) graduate courses available at a wide range of universities (see CES inventory of courses at www.evaluationcanada.ca) with potential for specialization in evaluation under related degree designations (e.g., community psychology, health sciences, educational administration). There exist some evaluation certificate programs (see discussion in Section 3.0 below) but these are relatively few and far between.
  • Cost - In order to establish a certification system Worthen (2003) suggests that four main challenges need to be addressed: (1) determine what the basic approach to certification in an individual's preparation might entail; (2) reach agreement on what core knowledge and skill ought to be possessed; (3) construct legally defensible certification procedures and instruments; and (4) garner support for mandatory certification process. In addition to these substantial start-up costs would be the costs of ongoing maintenance of an official registry and process to award certification. This would entail the creation of paid positions and the establishment of peer-review panels (Long & Kishchuk, 1997). Probably most problematic would be # 3 above, since agreed professional competencies would be required, and standards set to define minimum competencies for admission to the profession. Also required, Love (1994) reminds us, would be a valid professional code of ethics. A system installed to certify members would necessarily require a process for regulating the quality of program evaluation services delivered and de-certifying evaluators found to be in violation of the code of ethics. Implied would be significant association costs for malpractice insurance. All of these costs would be borne by members requiring certification, general membership of the society, partnering organizations or some combination thereof.
  • Grandparenting - When a professional society moves to a system of certification, what would be the implications for extant members of the society? Altschuld (1999b) points out that evaluation societies have been most concerned to date with ensuring their longevity through building up and maintaining decent membership roles. A requirement for certification might best be handled through a two-tiered system thereby not excluding membership in the professional society to interested parties who do not practice evaluation. There would be risks of false positives (members being certified who should not be) and false negatives (applicants being denied certification who should not have been). Altschuld is of the view that a grandparenting clause would have to be fairly generous and that a sufficient grace period may be required (personal communication, April 3, 2006).
  • Litigiousness of society - Worthen, once a strong advocate and proponent of certification of evaluators, has more recently softened his stance in the light of two primary considerations. The first, outlined above, is the evolving diversity of the field which, according to Worthen, has led to "enjoining tolerance and eclecticism [rather than] rallying around what we deem to be essential evaluation knowledge and skills." (2003; p. 340). The second is the ascendant litigiousness of society. "A tendency to address disputes, disagreements, and disappointments through litigation has reached near epidemic proportions in our society." (p. 340). No grievance or legal action can be taken lightly, regardless of its apparent laugh-ability or absurdity. And the time, energy and costs required to deal with legal challenges can be breathtaking.

The upshot of the forerunning discussions and deliberations has been that the field is not quite ready for individual-level certification in the form of either licensure (implying regulatory legislation and exigencies for a valid licence to practice) or certification by the professional society. A somewhat less ambitious proposal did emerge from the AEA task force, however; a proposal for a system of 'credentialing'. The strongest advocate for the approach has been Altschuld (1999b, 2005) who defines credentialing as "a set of courses or other experiences a person must go through to receive a credential. This may be done by a professional society or sometimes by trainers as in a credential for having been trained" (2005, p. 159). A credentialing system does not specify the skill set attained by the person who is credentialed, only that they have gone through delineated experiences and courses. This is consistent with Love's (1994) distinction between a professional development approach and a licensure approach to certification. Credentialing aligns with the professional development approach.

The credentialing option occupies middle ground between hard-nosed certification based on, for example, validated core competencies, and the status quo, which appears to be general adherence to principles of professional practice with no mechanism to ensure that practicing evaluators have had sufficient training or professional experience to enable full appreciation of or respect for them. Credentialing could be viewed as a transitory or intermediary step, one that might eventually lead to the development of a more stringent certification system. As Altschuld suggests:

[Credentialing] is not the same thing as certification but it would be a step in the right direction. The bottom line is that continuing as we have been without any control of entry into our field is not sensible for the long term.

(2005, p. 167)

Despite valiant efforts to move the AEA in this direction, including public endorsement of the proposal by then AEA president Bickman (1999), nothing materialized in the US, probably due to many of the challenges identified above, and the apparent muted enthusiasm of AEA members as revealed by the survey of the membership (Jones & Worthen, 1999). But in Canada, as Gussman (2005) aptly points out, current circumstances in the federal government sector provide a contextual backdrop in which such a proposal would be more likely to receive support. We applaud and support Gussman's advocacy of a voluntary credentialing system in Canada, while concurring with him and with Altschuld that a transitional period of grandparenting would run the risk of generating false positives (evaluators who are credentialed but should not be) and false negatives (evaluators who are not credentialed but should be).

At the basis of a credentialing system would be criteria for determining which experiences and training opportunities count and which do not. This in turn implies the need for a unified or agreed upon set of evaluator competencies. We now turn to an examination of the state of the art of work in this area.

 

2.3 Core Competencies for Evaluators

It is the primary goal of standards, principles and guidelines for professional practice in evaluation to enhance the quality of evaluation outputs and products. That is, they help to envisage what good quality evaluations might look like and provide general guidance for evaluation practitioners on how to produce them. But they do not delineate the evaluator competencies that would be necessary to produce good evaluations. The development of such competencies is essential to the professionalization of evaluation, as implied in the foregoing discussion on requirements for certification. But such development would also be useful for several other purposes, including advocacy for the field, professional development, and the development and assessment of evaluation training programs.

Stevahn, King, Ghere and Minnema (2005a) note that several interested authors and parties have developed lists of tasks and skills required to carry out effective evaluations over the years. In Canada, one such list was developed by CES and forms the curricular basis of the Essential Skills Series , a series of four intensive day-long workshops intended to introduce new evaluators to the basic principles of program evaluation including planning evaluations, using evaluation as a management tool to improve program performance, and evaluating for results (CES, n.d.-a; see also, Nagao, Kuji-Shikatani & Love, 2005). The Essential Skills Series reflects the professional development approach to certification developed by CES and recently was used as a basis for developing an accreditation pilot program for school evaluators by the Japanese Evaluation Society (Nagao et al., 2005). As another example, TBS developed Evaluation Standards for the Government of Canada as part of its Evaluation Policy with the goal of providing "departments with a basis for improving the quality of evaluation practice … and [providing] Treasury Board Secretariat with a basis for monitoring the implementation of its evaluation policy." (2001, Appendix B) The TBS standards provide definitions and guidance for standards associated with evaluation planning and issue development; evaluation competency; objectivity and integrity; consultation and advice; measurement and analysis; and reporting.

While such lists may have proven useful in providing guidance for training, policy monitoring and shaping evaluation practice in desirable ways, "none of the proposed frameworks appeared to be systematically derived or empirically validated through consensus building among diverse professionals across the field" (Stevahn et al., 2005a, p. 103) [1] . Recently, two important projects have been undertaken that address this shortcoming. The first has emerged from a group of American researchers led by King and Stevahn (King, Stevahn, Ghere & Minnema, 2001; Stevahn et al., 2005a, 2005b) and has resulted in an empirically validated set of 'Essential Competencies for Program Evaluators' (ECPE) that evolved over a five-year period and was revised on the basis of input from American and Canadian evaluators and experts in the field. These competencies are organized under six categories or themes:

  • professional practice (6 competencies): knowledgeable, ethical, professional practice;
  • systematic inquiry (20 competencies): technical skills to conduct evaluation;
  • situational analysis (12 competencies): understanding evaluation as a political activity;
  • project management (12 competencies): skills to manage evaluation projects;
  • reflective practice (5 competencies): ongoing professional development;
  • interpersonal competence (6 competencies): negotiation, conflict resolution and related skills; 
    (see Stevahn et al., 2005a).

A second set of competencies derive from another empirically grounded inquiry commissioned by the CES in support of its evaluation advocacy agenda. The Core Body of Knowledge (CBK) project was undertaken by a group of researches led by Zorzi (Zorzi, Perrin, McGuire, Long, & Lee, 2002; Zorzi, McGuire & Perrin, 2002; see also McGuire & Zorzi) and involved literature review, and consultation via the Internet with evaluation practitioners both within and outside of Canada about competencies required to complete a specific self-identified evaluation from the past. Thirty-six evaluation experts with diverse backgrounds interpreted the findings. The purpose of the project was to develop a core body of knowledge about program evaluation by identifying the benefits of evaluation, the outputs (e.g., evidence, conclusions, recommendations) of evaluation required to bring about those benefits and the knowledge and skills needed to produce such outputs. The result was a list of 23 general knowledge and skill elements, within which more specific knowledge, skills and practices were identified. Each element was categorized into one of the following clusters: ethics, evaluation planning and design, data collection, data analysis and interpretation, communication and interpersonal skills, and project management. The group concluded that the competencies needed to conduct an evaluation vary depending on the purpose and context of the evaluation; evaluation competencies are constantly evolving; and no individual need be competent in all areas.

The reader will notice considerable overlap of these elements with the ECPE framework developed by the King and Stevahn group. But added value from the CBK study is associated with its explicit attention to evaluation benefits (accountability, decision making, program-relevant knowledge and skill; social change; and cohesion and collaboration) and the outputs required to realize such impacts (various evaluation products, outputs of stakeholder involvement, and cross-cutting outputs). It is argued by the authors that explicit awareness of such benefits and means of achieving them would be enabling to evaluators' ability to carry out evaluations that will be effective in meeting stakeholder needs and achieving intended objectives and uses. This logic is consistent with an observed gap in the evaluation capacity building literature, namely that considerable research has been done on developing the capacity to do evaluation but comparatively little is known about developing the capacity to use evaluation (Compton, Baizerman & Stockdill, 2002; Cousins, Goh, Clark & Lee, 2004; Cousins et al., 2006). Perrin (2005) says it well,

… there appears to be increasing concern that the particular benefits that can accrue through evaluation (as opposed to, for example, through the work of auditors, or management consultants with rather different skill sets and approaches) are not always well understood …. It is also necessary to identify the outputs or benefits (outcomes) that users or clients of evaluation can obtain, and how competencies specific to evaluators can help bring these about.

(p. 175)

To conclude, it seems that considerable recent work on evaluator knowledge and skill competencies has been done and that this work might form the foundation of further concrete actions and strategies in the interest of enhancing evaluation quality assurance. There seems to be general agreement that it would be premature to use these competencies as a basis for developing a system of certifying individual evaluators - what Love (1994) calls the licensure approach to certification. Altschuld (2005), Stevahn et al. (2005a), Perrin (2005) appear to be in agreement about this. Yet, there is also general acknowledgement from these authors and others that competencies can be used to move an evaluation advocacy and quality assurance agenda forward. Perrin (2005), for example, discusses several uses of competences: basic education and training, self-evaluation, professional development, information and advocacy about the skills needed to do competent evaluation, and as a tool to assist those who commission evaluation. He supports the use of competency information to inform, rather than to prescribe, particularly since evaluation is considered to be a transdiscipline and because of the many unresolved issues outlined above.

Stevahn et al. (2005b) argue that competencies ought to be used as a basis for designing and developing university-based training for evaluators. They propose that a consortium of interested parties might convene to forge a set of standards for evaluation training programs. This would include a cross-walk of sorts across various sets of evaluator competencies that have been developed to date. Envisioned is a voluntary accreditation system for organizations offering evaluation training. Accreditation would imply that the program in question meets the agreed set of standards. While there might be some merit in this argument, Altschuld provides a counter perspective based in part on his own involvement in reviewing available programs for evaluation training (Altschuld et al., 1994; Engle et al., in press). The essence of his concern is that, 

…when the skills and competencies are reviewed, it is apparent that not a single evaluation training program at a university or institute could produce an individual with all the prerequisite skills, competency levels and experiences necessary for an accomplished evaluator…

(p. 2005, p. 166)

Most programs, according to Altschuld (2005) are quite small, involving 2 or 3 evaluation content courses. They are diverse, tend to be specialized within areas, and may have varying purposes and objectives. Yet, he sees value in evaluator competencies in supporting the development of an evaluator credentialing system, as described above. Credentialing, supported by a unified set of competencies could lead to the development of evaluation programs across disciplines. Further, differentiated credentials could be offered for beginning levels and higher levels of competency, depending on the training and professional experiences undertaken. Competencies could also be used to determine the fit of evaluation training programs and other professional and practical experiences with what is deemed as important evaluation practice. 

 

2.4 Implications for Professionalization of Evaluation in Canada

In this review and integration of literature we explored three separate yet inter-related themes. First we examined the relationship between evaluation and government giving particular consideration to developments in the Canadian context such as evaluation's role within the contemporary over-arching management framework. Next we examined in detail the main arguments associated with the professionalization debates which have taken place over the years. Finally, we reviewed progress toward the development of evaluator competencies and the implications of such work.

We conclude that there is a clear role for government in developing the professionalization of evaluation and enhancing quality assurance in evaluation within government and beyond. The state of development of professional practice in evaluation is such that it would be premature at this time to set up a professional certification process that would be oriented toward the licensure option depicted by Love (1994). Rather, it seems prudent to continue with efforts within the professional development approach that would capitalize on the fine work on the development of evaluator competencies that has been completed to date. Implicated would be government support for the development of a credentialing system that would provide some basis for deciding who has sufficient background training and experience to conduct evaluations of high quality. This in turn would imply the need for greater involvement and collaboration between government and the university sector on the one hand, and with the professional society (i.e., CES) on the other.

A credentialing system as described above would imply that university-based and other suitable training programs are available in Canada. We now turn to an examination of the current state of university-based evaluation training in Canada and how that compares with what is available in other jurisdictions. To follow that section will be a review of university-based centres of excellence in Canada and beyond. Of particular interest is the extent to which and how such centres can contribute quality assurance in evaluation in the government sector.

 

3. Survey of University-based Evaluation Training Programs and Options

In this section we survey university-based evaluation training programs outside of and within Canada. First we examine programs located beyond Canadian borders, mostly in the US. This provides a basis for comparison of the university-level training available in Canada, which is the focus of the next section. Finally, we examine in greater depth a joint government-university program offered in the UK. This program is profiled because of its unique character, particularly with respect to the role of government in evaluation quality assurance.

3.1 Evaluation Training beyond Canadian Borders

3.1.1 University-based programs
The vast majority of university-based evaluation training programs that we identified are located in the US. Our primary source for identifying these programs was Altschuld et al. (1994) who conducted a survey of evaluation programs. A prior survey had been carried out by May et al. (1986) and this was used as a basis of comparison by Altschuld and associates. More recently Engle and Altchuld (2003, in press) updated their survey published in 1994. In both cases they defined a program to imply "multiple courses, seminars, practica, offerings, and so on designed to teach what the respondent considered to be evaluation principles and concepts." (Altschuld et al. 1994, p. 72).

The authors note a decline in evaluation programs over the years. At the time of the prior survey (1994), there were 49 programs, 38 in the US (down from 44 reported by May et al. (1986). "It is important to note that of these 49 programs, only on half (25) had the word "evaluation" in their official title, limiting the visibility of the others" (Engle & Altschuld, 2003, p. 13). In the most recent survey, the authors reported a total number of programs has decreased from 49 to 36 (26 in US). The authors speculate that this may be because senior evaluation leaders are retiring. "We have not yet begun to see the next generation of university-based programs led by passionate young faculty." (p. 13). Again, visibility is a key issue, with 22 (61%) having the word 'evaluation' in their title.

These surveys provided a starting point for our own internet search for programs. Specific details about programs appeared in the 1994 survey report, but no such information was reported recently. We therefore selected programs that appeared to bear some relationship to government (usually in the form of internship or practicum placements) and then attempted to locate current web pages. Some of these programs no longer exist, while others struck us to be less programs than one or two courses in evaluation that would be integrated into existing master's or doctoral studies. The program at the University of California Los Angeles in Social Research Methodology is a good case in point (M. Alkin, personal communication, April 11, 2006). We report only those programs where an overt specialization in evaluation could be obtained. Our final sample is summarized in Table 2. Appendix A lists all programs (outside of Canada and within) that we located with more detailed information on each [2] . It should be noted that our web survey was limited in that information is variably reported and explicit connections to government are not very often in evidence. The final sample is not exhaustive but we believe it to be representative of the state of the art of evaluation training at universities.

Table 2 provides a useful snapshot of what is currently available in terms of university-based evaluation training programs outside of Canada. We only located two programs outside of the US, one at the University of Melbourne in Australia, and the other in the UK. The latter program is a joint initiative between the Institute of Education, University of London and government and it is profiled in more detail in the Section to follow. We confirmed the program in Melbourne to be the only one available in Australasia (J. Owen, personal communication, April 7, 2006).

Table 2: University or Government-based Evaluation Training Programs

Name Location

Mission 

Credentials Offered Human Resources Practicum / Internships
American University
School of Public Affairs
Department of Public Administration & Policy
USA
Washington, DC
Training of practitioners, researchers / scholars Graduate Certificate in Public Policy Analysis, MPA, MA in Public Policy; Ph.D. in Public Admin: Evaluation-related courses offered 24 Faculty members within School Internships and co-ops with employers in the public sector.
Ball State University Department Of Educational Leadership USA
Muncie, IN
Training of practitioners, researchers / scholars MA; Ed.D. in Educational Administration and SupervisionEvaluation courses offered as part of program 6 Faculty within department Internships are part of the program but not necessarily in evaluation.
Claremont Graduate School
School of Behavior and Organizational Sciences
USA 
Claremont, CA
Training of practitioners, researchers / scholars Ph.D, M.A. in evaluation;Evaluation Certificate Professional Development, summer workshops 12 Faculty members, 5 teaching directly in evaluation programs Yes, internship courses and assistantship practical opportunities.Placement course offered at Ph.D.
Florida State University, Department Of Educational Research And Learning Systems USA
Tallahassee, FL
Training of practitioners, researchers / scholars Graduate Certificate in Program Evaluation; Measurement and Stats, Ed. Accountabilty; Ph.D., M.A, in Meas. and Stats, eval. courses available 11 Faculty within department None mentioned.
Institute of Education, University of London, Government Social Research Unit (GSRU), Cabinet Office UK
London
Training of practitioners M.Sc in policy analysis and evaluation 13 Faculty from Cabinet Office, Institute of Education, University of London and GSRU No practicum mentioned.
Iowa State University, Department of Educational Leadership & Policy Studies USA
Iowa City, IO
Training of practitioners, researchers / scholars M.Sc.as Ph.D. with concentration in research and evaluation; Eval. courses also in Agricultural Ed Dept 3 Faculty in evaluation program None indicated.
Michigan State University, Department of Psychology

USA
East Lansing, MI

Training of practitioners, researchers / scholars Ph.D. program: Graduate concentration in Quant. Methodology and Evaluation Science 8 Faculty; inter-departmental program None indicated.
Tufts University, Graduate and Professional Studies

USA
Medford MA

Training of practitioners Graduate Certificate in Evaluation (15 credits) Not indicated Practicum course required.
University of California, Berkeley, Graduate School Of Education

USA
San Francisco, CA

Training of practitioners, researchers / scholars Ph.D. in Quantitative Methods and Evaluation Ed.D. or Ph.D. in Evaluation and Assessment Training 11 in Policy, Organization, Measurement, and Evaluation group Students participate in evaluation and/or assessment apprenticeships in ongoing assessment and evaluation projects. Research assistantships are available.
University of Illinois, Urbana-Champaign Department of Educational Psychology USA
Urbana- Champaign, IL
Training or researchers/ scholars Ph.D, MA in Education Psych; Evaluation and research focus 15 Faculty members with specialization in quant methods and stats, evaluation, measurement No mention of practicum courses or placement opportunities.
University of Maryland, Department of Measurement, Statistics & Evaluation USA
Baltimore, MD
Training of practitioners, researchers / scholars M.A.; Ph.D. ; Graduate Certificate in Measurement, Statistics and Evaluation 11 Faculty located in department, within Faculty of Education No practicum mentioned. Current and recent students have conducted special projects with or have been employed by a wide variety of organizations engaged in evaluation.
University of Melbourne
Faculty of Education Centre for Program Evaluation
AUS
Melbourne
Training of practitioners M.A. with specialization in evaluation; Graduate Certificate in Evaluation (12 credits) 9 Faculty teaching in program from 2 Centres (Program evaluation; assessment research) Assessment Yes, evaluation project available as part of both programs.
University of Minnesota
College of Education and Human Development
USA
Minneapolis, MN
Training of practitioners, researchers / scholars Ph.D; M.A.in evaluation Evaluation Certificate (graduate degree required for admission) MESI evaluation institute 12 Faculty members 
3 adjunct faculty members
Yes, internship courses and assistantship practical opportunities. Link to CAREI Internship course offered at M.A, Ph.D. and certificate level.
University of Virginia
Curry School of Education, Department of Leadership, Foundations and Policy
USA
Charlottesville VA
Training of practitioners, researchers / scholars M.Ed. with emphasis in Research, Statistics, and Evaluation; Ed. D.; Ph.D. with emphasis in Research, Statistics, and Evaluation Not indicated Ed.D, Ph.D. - practicum or internship work.
Western Michigan University
Faculty of Education
USA
Kalamazoo MI
Training of practitioners, researchers / scholars Ph.D; M.A. in evaluation, measurement and research 5 Faculty members dedicated to Evaluation, Measurement and Research Yes, evaluation practicum and professional field placements are part of the Ph.D. program.

In many cases evaluation programs are offered as a specific focus within an existing graduate degree, either at the master's or doctoral level. Several of these involved the study of quantitative methods, measurement or statistics. In some cases (e.g., Claremont Graduate University, Western Michigan University, University of Minnesota) degrees specifically in evaluation are offered. Ten out of the seventeen programs that we profiled are offered through Faculties or Colleges of Education. Unlike Canada, education in the US is a federal matter, and standard evaluation texts observe that much of the early work done in evaluation of focused on the evaluation federal nation-wide educational programs. Other locations for programs include several schools or departments of psychology and in the case of the American University, a School of Public Affairs. This was the only program located in such a school. The Ph. D. program at Western Michigan University is in fact, an interdisciplinary program involving four colleges at the University (A. Gullickson, personal communication, April 11, 2006; Stufflebeam, 2001). According to Stufflebeam (2001) this program took lead from Stanford Evaluation Consortium headed up by L. Cronbach: the ideal evaluation degree program would include (1) disciplinary preparation in social sciences, (2) participation in interdisciplinary seminars that examine evaluations, (3) apprenticeship to practicing evaluators (preparation of critiques, assist in drafting proposals, interpretation of data, communication of findings), and (4) an internship at an agency were policy is formulated. In programs that were clearly identifiable as evaluation training programs, we did not find explicit evidence that the curriculum is based on an identified set of evaluator competencies, a route toward quality assurance proposed by Stevahn et al. (2005b).

In addition to degree programs, several universities (7 or 41%) offer graduate certificate programs in evaluation, usually involving about 15 credits of study (5 courses). In the case of University of Minnesota, the certificate is only available to those holding a graduate degree, but such is not the case anywhere else. Usually the certificate is offered at the master's level and in most cases, credits from the certificate could subsequently be applied toward a graduate degree. Melbourne offers the certificate program through distance mode. This program was recently offered under contract to the Government of the Northwest Territories in partial distance, partial onsite delivery mode (J. Owen, personal communication, April 7, 2006.)

The number of full time regular professors teaching in the evaluation program was often difficult to determine. In many cases, the only information available was the number of faculty as full time members of the department or academic unit. Nevertheless, programs that did specify faculty teaching in the evaluation program, typically identified at least 5 members.

Finally, there was a mix of programs offering practica and internships and those offering course and thesis work only. In the case of the former, it was usually explicitly stated that the development of effective practitioners of evaluation or policy analysis was part of the mission of the academic unit. It was sometimes the case that professional development activities were offered as well as formal certificate or degree programs (e.g., Claremont Graduate University, University of Minnesota). In some cases the mission is explicitly to develop potential researchers and scholars in evaluation. Usually such programs offered Ph.D. degrees. 

3.1.2 Other training options
Identified in the review of literature is the phenomenon of the unpatterned career path: We know that many evaluators enter this domain of professional practice through circuitous routes. This does not necessarily imply that they enter the field with no training or that training and professional development is not available to them on entry. Some universities identified in Table 2, for example, offer in addition to degree and certificate programs professional development institutes and short courses. Claremont Graduate University and University of Minnesota are two examples Additionally, topic-specific workshops and short courses are available at national and regional conferences such as the Australaisian Evaluation Society, the European Evaluation Society and the American Evaluation Association. In the US, the 'Evaluator's Institute' provides another option. This privately-run organization hosts an annual institute that brings together many well known contributors to the field for workshop sessions. Such options, however, might be most appropriately thought of as in-service activities designed to augment evaluator knowledge and skills, rather than pre-service preparatory experiences.

In addition to these training and professional development opportunities there are two training options that potentially contribute to quality assurance in evaluation. The first is government-directed training and the second is enrolment in the single university-based course on evaluation. Both options might involve certification of achievement rather than participation. In other words, unlike many professional development institutes and workshops, successful achievement of knowledge and skills would be required to pass the course(s). We now turn to a discussion of such training options for evaluators.

In 1998 General Accounting Office (GAO) of the US federal government established its own Training Institute to separate training and education from career counseling and personnel and organizational development support. According to Kinsgbury and Hedrick (1994) the Training Institute was deemed critical to meeting the information needs of the US Congress. It was comprised of 17 classrooms in a Washington DC location plus training facilities in all regional offices. In order to continue to be deemed qualified to do the GAO's audit and evaluation work, every evaluator must obtain a minimum 80 hours of training every two years. The Institute offered six focal areas of study for evaluator training: mission, policies and individual responsibilities; assignment planning and evaluation; communication; computer use; workplace relations and management; and issue area training. Also self-paced training was available in the form of internet courses. In their critique of the program Kingsbury and Hedricks (1994) concluded that matching training to job relevance is critical, involving both line managers and staff in training increases its credibility and impact, training needs to deliver consistent messages at all levels and there is an ongoing need to evaluate the training programs in order to assure quality. As mentioned above, the Training Institute has now evolved to the Center for Learning, but it continues to be governed by Government audit standards. Even with its emphasis on internal training, the US GAO has recently consciously altered its hiring practices to focus on highly trained personnel, typically at the Ph.D. level. Some such individuals may have been trained in specialized evaluation programs but this is not necessarily the case (N. Kingsbury, personal communication, April 7, 2006).

Even in the case where government-level evaluation training is available, the single university-level evaluation course represents another option adopted by many practicing evaluators. Several of the university evaluation programs identified in the Altschuld et al. (1994) survey were not included in our own review in Table 2 because our survey of their websites revealed that only one or two evaluation courses may have been offered, rather than a set of courses contributing to a program. Yet, such courses provide aspiring and practicing evaluators with worthwhile options for intensive study in evaluation, normally about 40 hours or so (3 credits) at the graduate level. Candidates might choose to do the single evaluation course as part of a related degree program such as education and psychology. Morris (1994) observed that this scenario is probable since only 11 percent of AEA members indicated training in formal evaluation degree and certificate programs in the 1992 membership directory. Another option would be to enroll in such a course as on a special student basis. Candidates must be eligible for study at the graduate level in order to enroll for single courses of this sort (e.g., upper second class standing in relevant bachelor's degree).

Morris (1994) provided an in-depth examination of the single evaluation course phenomena, giving explicit consideration to the role of such courses and what they might include. He sees the single course as a valued part of professional training, not as a cut-and-dried competency development exercise. Yet, its major contribution would be to the production of educated consumers as opposed to competent practitioners. But that, says Morris, can be a good thing, particularly in the light of the need to educate non-evaluators about the power and virtue of the field. "Although a little knowledge can be a dangerous thing, program evaluation is a field in which total ignorance is much worse. Evaluation is most likely to achieve its dual goals of demonstrating scientific credibility and bettering the human condition in an environment where it is not just the professional evaluation community that has access to relevant knowledge and skills" (1994, p. 57). Clearly, however, something deeper and more intensive is required in order to develop evaluator competencies and to inform practice.

3.1.3 Summary
In summary, our survey of training programs abroad revealed that most identifiable university-based evaluation programs are located in American universities (exceptions in Melbourne and London) and that these may be somewhat on the decline, due perhaps to the point in the career trajectory of the founding members of the programs. Nevertheless, we located several graduate degree and certificate programs in evaluation. There exist many options for ongoing professional development and continuing education in evaluation but most of these do not lead to formal certification of achievement, as opposed to participation. Exceptions would be the Training Institute run by the US GAO and single evaluation graduate courses offered in many university departments and faculties. We now turn our attention to the state of the art of university-based training in Canada.

3.2 State of the Art of Evaluation Training in Canada

3.2.1 University-based and post-secondary programs
Despite some Canadian universities being identified in the Altschuld and Engle surveys (1994, 2003) degree programs in evaluation at the masters and doctoral levels do not currently exist in Canada, although in some cases doctoral degrees with specialization in measurement and evaluation do exist. To our knowledge these are typically located in faculties of education and take up as a central concern large scale assessment and standardized measure of academic achievement (e.g., Université Laval, University of Ottawa, University of British Columbia). A very long list of Canadian universities, however, offer graduate courses in program evaluation, as evidenced by the updated list provided by the CES (see www.evaluationcanada.ca). It is entirely likely that one would be able to specialize in evaluation as part of a degree program associated with a related area of inquiry (e.g., educational administration, community psychology, health sciences, criminal justice).

We profiled a number of Canadian post-secondary programs (all, except for Georgian College, located at universities) in Table 3 for comparative purposes. To identify the sample we started with the Canadian programs included in the Altschuld et al. (1994) survey and verified their web-sites to ensure that at least two evaluation-related courses were available within the same degree program or concentration area. Our rationale was that the availability of at least two courses provides an opportunity for specialization in evaluation as part of some related disciplinary program (the specialization would not be likely to be formal or to appear on the degree, yet it would be possible for candidates to pursue thesis work in this domain). Some universities previously identified by the Altschuld group (e.g., University of Calgary, University of Lethbridge) were eliminated for failing to meet this criterion. Other universities were added. Our resource for locating them was the CES list of post-secondary courses mentioned above. We examined websites of universities suggested to offer two or more evaluation courses within a single faculty or department. Finally, a program at Georgian College, a community college located in Ontario, was identified by virtue of its success in competing in the CES Student Case Competition in 2005. Our final sample identified 12 Canadian universities and one community college (see Table 3).

Table 3: Post-Secondary Evaluation Training Options in Canada

Name Location

Mission 

Credentials Offered Human Resources Practicum / Internships
Carleton University
School of Public Management
Ottawa, ON Training of practitioners, researchers / scholars MA
Ph.D.
Diploma in public administration
Evaluation courses also available in International development, social work, psychology
21 Faculty teaching in the school
4 professors emeriti
11 adjunct professor
No practical component in evaluation mentioned. Practical component in public service management available in masters program. 
Dalhousie University, School of Public Administration Halifax, NS Training of practitioners MPA
MPA with various specializations (mgt, law, library sciences)
10 Faculty members associated with School of Public Admin
8 Adjunct Faculty
No practicum mentioned.
École Nationale d'administration publique Montréal, Québec, Gatineau, PQ  Training of practitioners, researchers / scholars Ph.D; M.P.A. with courses in evaluation;
Graduate Certificate in program evaluation (15 cr.)
4 Faculty members in program evaluation;
1 faculty member in performance measurement
No mention of practicum requirement.
Situated within public administration
Georgian College, Research Analyst Program Barrie, ON Training of practitioners Diploma (3 semesters) 5 Faculty
3 associated faculty
23 advisory committee members
Yes, internship placement for third semester.
University of Guelph Department of Psychology Guelph, ON Training of practitioners, researchers / scholars Ph.D Applied Social Psychology 
program evaluation-related courses offered
8 Faculty members Placement opportunities and consulting work with all levels of government
Université de Laval
École de psychologie
Québec, PQ Training of practitioners, researchers / scholars Ph.D. Recherche et intervention (orientation communautaire). Cours en évaluation de programme, évaluation psychosociale des milieux, consultation et gestion dans la pratique professionnel Nombres de professeurs associés avec le programme non spécifié  Stages qui se centrent en évaluation, consultation, gestion, ou supervision. Lieux des stages ne sont pas spécifiés
University of Ottawa, Faculties of Education and Social Sciences

Ottawa, ON

Training of practitioners, researchers / scholars Graduate Certificate in Evaluation (pending)
M.A., Ph.D. Educational Measurement and Evaluation; Clinical Psychology
6 Faculty in education and psychology teach evaluation Practicum training and internship placement in evaluation offered through School of Psychology at Centre for Research on Community Services. Placements generally in community/social services and governmental contexts.
Université de Québec a Montréal, 
Faculté d'éducation et Département de psychologie

Montréal, PQ

Training of practitioners Programme court de deuxième cycle en évaluation de programmes d'éducation et de formation (15 cr.). 
Psy.D. psychologies (profil professionnel) ; Ph.D. psychologie (profil scientifique-profesionnel). Cours en méthode de recherche en intervention (6 crédits)
Not indicated Intégration workshop in évaluation (1 cr.) 
Stages d'évaluation (15 cr.), processus psychologique et d'évaluation - approche systémique / social (3 cr.)
University of Saskatchewan
Applied Social Psychology Program

Saskatoon, SK

Training of practitioners, researchers / scholars Ph.D.; M.A. in Applied Social Psychology, includes courses in evaluation
Evaluation courses also offered in Faculty of Ed.
5 Faculty members associated with Applied Social Psychology Program Practicum in evaluation
Internship in evaluation
University of Victoria
School of Public Administation

Victoria, BC

Training of practitioners, researchers / scholars M.PA; Ph. D. or diploma programs.
Graduate certificate program in performance management incl. eval. and performance measurement as a core course. (12 credits)
5 Faculty members teach in the School Evaluation practicum and field experience courses can be part of the Ph.D. degree.
U of Waterloo
Health Studies and Gerontology

Waterloo, ON

Training of practitioners, researchers / scholars Ph.D, M.Sc. 
Evaluation courses can be done as part of the degree programs
33 Regular Faculty, 17 adjunct members of the Health Studies and Gerontology Program No practicum mentioned
U of Windsor
Department of Psychology

Windsor, ON

Training of practitioners, researchers / scholars M.A. and Ph.D. programs in Applied Social Psychology: Program evaluation, research methods and measurement courses offered 9 Faculty members Practica and internships in government agencies, community organizations, schools / colleges
Provincial research funding evident
Wilfrid Laurier University, 
Department of Psychology

Waterloo, ON

Training of practitioners, researchers/ scholars M.A. and Ph.D. 
programs in Community Psychology Courses on research in community settings at M.A. level and on program evaluation and community research and action at Ph.D. level
Number of Faculty members dedicated to community psychology area not specified Practica in community settings with students receiving training in consultation, program development, program planning, and program evaluation.

While there are no degree programs offered in evaluation in Canada, we did locate three graduate certificate programs in evaluation. Two of these are located at French-language universities, one in public administration (École nationale d'administration publique), the other specifically devoted to education (Université de Québec à Montreàl). A third program is structured as a bilingual, joint Faculty of Social Science and the Faculty of Education program and, pending final approval, will be offered starting fall 2006 at the University of Ottawa. In each case, the graduate certificate requirement is 15 credits (5 courses) completed on a full- or part-time basis, which is very much consistent with what we observed in programs beyond Canadian borders.

Evaluation courses and the potential for specialization in evaluation under an associated discipline are offered at several universities under various disciplines including Applied Social Psychology or Community Psychology (e.g., Université de Laval, University of Guelph, University of Saskatchewan, University of Windsor, Wilfrid Laurier University), Education (University of Ottawa, Université de Québec a Montréal), Health Studies and Genentology (University of Waterloo) and Public Administration (Carleton University, Dalhousie University, École national d'administration publique, University of Victoria).

As was the case with our international survey of programs, it is difficult to determine how many faculty members are teaching evaluation courses in each program, but certificate programs appear to be staffed by a minimum of five faculty members.

Practical experiences in evaluation are mostly associated with evaluation certificate and diploma programs, but also with some universities currently not offering a certificate (e.g., Université Laval, University of Saskatchewan, University of Victoria, University of Windsor, Wilfred Laurier University). Practical experiences range from internship opportunities, such as those offered by the University of Saskatchewan, to practicum courses where candidates participate in practical evaluation work under the supervision of university instructors (e.g., University of Ottawa, University of Victoria).

3.2.2 Other training options
As would be the case in the US and elsewhere, candidates may actually enrol in graduate-level university courses in evaluation and take them on a special student basis. Such persons are required to be eligible for graduate study at the various universities (e.g., bachelors degree with second class standing) and typically they would be permitted to complete only two such courses on a special student basis. The advantage to following such courses is that they are recorded as achievements on the respective university transcript, they confirm candidate eligibility for graduate study, and in many instances, they may subsequently be applied toward university degree or certificate programs (under such conditions as program requirements and recentness of completion). The list available at the CES website (www.evaluationcanada.ca ) shows the breadth of universities offering such courses across the country.

Continuing with post-secondary options in Canada, the Research Analyst Program at Georgian College (see Table 3) is not a university-based program but candidates are expected to have at least three years of post-secondary education prior to admission. The program consists of 12 primarily skill-building courses in various research-oriented topics and an internship is required for the third and final semester. This program represents an alternative to university-level study that results in a diploma. A variety of other training options are available in Canada for those interested in developing knowledge and skill in evaluation, but the majority of these might best be thought of as professional development activities that result in a certificate of participation rather than credentials of achievement such as credit courses or certificates that are recorded on university transcripts.

In Canada, the Canadian School of Public Service provides internal training nation-wide for Canadian federal employees, and periodically offers courses on evaluation. Such courses are considered to result in credentials of achievement. Also offered on a regular basis by the CEE are a series of topical workshops available to public servants with an interest in evaluation. These opportunities are similar to the Essential Skills Series and intermediate level training opportunities (i.e., programs in survey design and logic models) offered by CES, in as much as they result in a certificate of participation, rather than a credential of achievement. The ESS certificate is awarded after participation in four day-long modules (CES, n.d.-a; see also, Nagao, Kuji-Shikatani & Love, 2005). The CES also sponsors workshops and professional development activities at the chapter level as well as annually at the CES national conference. Finally, universities occasionally hold such opportunities as summer institutes in evaluation and applied research methods although these do not appear to be offered on a regular basis such as we have observed elsewhere (Claremont Graduate School; University of Minnesota).

Despite the existence of these opportunities, in contrast to the US, for example, the Canadian federal government does not currently require that public servants employed in evaluation and policy analysis related jobs hold attestation that they have undertaken such training (see discussion of the Training Institute above).

3.2.3 Summary
Evaluation training opportunities in Canada are widely available but opportunities for advanced level university training appear to be quite limited. There currently exist no degree programs in program evaluation in the country and we located only three graduate certificate programs (one is pending approval) and one diploma program at a community college. While a wide array of universities offer graduate study in evaluation, this is most often limited to course-level experiences. Such courses may be integrated into degree programs (concurrently or subsequently) and it is likely that candidates could specialize in evaluation in degree programs in related disciplines such as education or applied social psychology. It is encouraging to note, however, that university courses in evaluation and related topics exist on such a broad basis and that several universities offer more than one evaluation course within single faculties or departments. The potential for the development, of certificate programs for example, would be increased in circumstances where faculties or departments could build on existing courses rather than developing programs from scratch. Finally, a wide variety of other training and professional development opportunities in evaluation exist both inside government and out, but at present there are no regulations requiring candidates to have undergone such training in order to hold evaluation-related posts within the federal government. We now examine an M.Sc. program in the UK that represents a partnership between the federal government and a university. This program is quite unique and bears quite directly on considerations of government's role in fostering evaluation quality assurance.

 

3.3 Case Profile of a Government/University Collaborative Program

The M.Sc in Policy Analysis and Evaluation developed in the UK and co-sponsored by the Government Social Research Unit, Cabinet Office and Institute for Education, University of London (2005) provides a model of institutional collaboration in the interests of evaluation quality assurance in government. The program is in essence a modular degree to be completed part-time over two years and began in October 2005, and it offers a unique opportunity to government social researchers looking to enhance their professional skills and career prospects. (http://www.gsr.gov.uk/professional_development/msc/index.asp ) The aims of the program are:

  1. to provide a broadly based training in social science research methods, with an emphasis on quantitative methods most relevant to government social science research staff;
  2. to develop students' knowledge and understanding of research design, research management, evidence-based policy, economic analysis, longitudinal analysis and a range of quantitative, evaluative and qualitative research methods; and
  3. to provide students with transferable skills which can be used effectively in different work environments

The program is designed to provide students with an understanding of the major quantitative research skills relevant to designing, analyzing and evaluating government policy. Participants are expected to gain a high level of critical insight into a range of research methods and to apply their understanding to policy and research questions and communicate their understanding clearly to both academic specialists in research and non-specialists. In so doing students would develop their existing skills in critiquing and applying research methods. The program was actually the brainchild of the Cabinet Office, Government Social Research Unit (GSRU) and built on a series of courses already developed for government researchers. The GSRU was interested in raising the quality of government social research and had some evidence that university programs did not provide people with the research skills that were needed in government (R. Taylor, personal communication, April 7, 2006).

The two year modular program structure, laid out in Table 4, consists of 5 compulsory and 2 optional modular credits. The program would be taken over a total of 24-26 days within a two year period and during regular working hours within the public service. The program is collaboratively delivered between Cabinet Office and Institute of Education but the Institute assumes responsibility for the assessment of the program, drawing up and marking assignments, establishing a board of examiners and appointing an external examiner.

Table 4: Modular Program Structure of the MSc in Policy Analysis and Program Evaluation (adapted from GSRU, 2005)

Modular elements  Module provider a Credit equivalents
Compulsory modules
Research & research management CO 1.5
Statistical analysis CO 3.0
Experimental & quasi experimental design CO 1.5
Research synthesis for policy and practice IoE/CO 3.0
Report (10,000 words) IoE 3.0
Optional modules (2 required)
Sampling design & data collection CO 3.0
Qualitative research & analysis  IoE 3.0
Economic & econometric analysis IoE 3.0
Longitudinal research & analysis IoE 3.0
a CO = Cabinet Office; IoE = Institute of Education, University of London

Compulsory modules focus quite heavily on quantitative methodology for social research including methods, statistics and design. There is also a module on evidence-based policy, or synthesis of research for policy and practice. Optional modules can be taken to extend methods, analysis and design capabilities including courses on qualitative methods and econometrics.

The program teaching team comprises eight Institute of Education faculty members, four Cabinet Office staff members and one consultant. Students of the program become members of the Institute of Education's Doctoral School and the Bedford Group of 'Lifecourse and Statistical Studies', another one of the Institute's Schools. The Bedford Group provides the teaching for the Institute's Doctoral School courses in statistics, multivariate analyses and survey methods. It is also responsible for course leadership for a MA and MSc in the Economics of Education and a new module on Quantitative Evaluation Methodology. In 2004/05 there were approximately 30 doctoral students registered in the Bedford Group.

The collaborative MSc in policy analysis and evaluation represents the first of its kind to our knowledge; a degree program jointly offered by a university and governmental organization, exclusively for members of the governmental organization. The program meets academic standards for the degree by virtue of affording responsibility for marking and candidate examinations to the Institute for Education, which is part of a chartered degree granting institution. While the academic standards of the degree speak to program quality, the heavy involvement of government in the design and delivery of compulsory and optional modules ensure relevance of the program to the policy analysis and evaluation exigencies of government. The program represents a clearly hands-on role for government in enhancing quality assurance in evaluation.

Table 5: Centres of Excellence in Evaluation and Related Fields

Name Location Mission a   Practice /Consultancy Human Resources Sustainability / 
Link to Government
Claremont Graduate University,
Inst of Org & Program Evaluation Research,
School of Behavioral and Org Studies
USA
Claremont, CA
Research
Practice
Training
Teaching
Program design, devel and eval; consultation in design, proposal prep; evaluation in HR, organizations; needs assessments, organizational and management consulting 6 Faculty
1 staff
students on project basis
Fees charged to students and clients for workshops and degree/certificate programs
Service provider relationship
Sponsorship unclear: mention of 'generous donations' for fellowships. 
George Washington University 
Center for Equity and Excellence in Education
Grad School of Ed and Human Development
USA
Arlington, VA
Training
Research
Practice
Conducts national and local ed. policy and applied research Designs and conducts program evaluation for states, districts, and schools, and analyzes policy. 2 admin: director, assistant director,
14 research scientists and associates, 
3 staff
Funding sponsorship unclear
Service provider relationship with state education agencies, local education agencies, and various offices of the U.S. Office of Education
Harvard University, Harvard Family Research Project (HFRP), Graduate School of Education USA
Cambridge, MA
Research
Teaching
Practice
Evaluation of varied initiatives for foundations, non-profit organizations and public agencies, family involvement in education; dissemination of research and theory on eval. 16 Faculty and staff members,
graduate and undergraduate student assistants.
Funding sources include: private foundations and corporations, public agencies
Indiana University, Centre for Evaluation and Education Policy, School of Education USA
Bloomington, IN
Research
Practice
Training
Evaluation literacy, education policy research and technical assistance; health, human services, and community development; math, science, and technology 5 management and academic staff
5 Faculty associates
6 research staff
Service provider relationship with state, regional and national governmental agencies and institutions, educational institutions and community organizations
Massey University,
Center for Public Policy Evaluation
College of Business
NEW ZEALAND
Palmerston North
Research
Practice
Evaluation in law and economics, health, economics and education and family Unclear Unclear: Likely sponsorship from government.
RMIT University;
Collaborative Institute for Research, Consulting and Learning in Evaluation
AUSTRALIA,
Melbourne
(Teaching)
Training
Practice
Program evaluation in education, community development, labour policy, health; implementation and outcome evaluation, performance measurement 3 Faculty members,
4 research staff, 
staff
postgraduate students
Fees to students for short course and certificates
Service provider relationship with Australian state and federal public agencies and New Zealand federal government
University Of Aberdeen,
Health Economics Research Unit,
College of Life Sciences and Medicine

UK
Aberdeen, Scotland

Research
Practice
Teaching
Training
Economic evaluation;
behaviour, performance and organisation of care;
evaluation of health improvement; valuation & implementation programme
22 researchers including Faculty members and research fellows,
support staff, 
Ph.D. students
Sponsorship: Chief Scientist Office of the Scottish Executive Health Department, competitive research grants
Service provider relationships: public agencies.
University of California Los Angeles 
Center for Research on Evaluation, Standards, and Student Testing,
Grad school of Education and Info St.

USA
Los Angeles, CA

Research
Practice
(Training)
Conducts major program evaluations, research-based assessments, technology as assessment tool, aid to schools and districts respond to the many accountability demands 4 Faculty members Partnership/consortium with 4 American universities, 1 UK university and Educational Testing Service.
University Of East Anglia,
Centre for Applied Research in Education
School of Education and Professional Devel.

UK
Norwich, England

Teaching
Research
Practice
Applied research including action research; programme and policy evaluation; consultancy; methodological development; research training; research degrees 16 Faculty members and researchers, 
8 visiting fellows, support staff, 
students
Sponsorship from European Commission, local and central governments, foreign national and state/provincial governments
Service provider relationships unclear.
University Of Illinois At Urbana-Champaign Center for Instructional Research and Curriculum Evaluation
College of Education; Dept of Psychology

USA
Urbana-Champaign, IL

Research
Teaching
Practice
Evaluation of programs in schools, education-related social services; technical and philosophical review of eval. projects, examination of questions of validity, utility of findings and ethical issues. Unclear: Faculty, associated Faculty and students Service provider relationship with schools and communities, state and federal programs, professional associations, and others.
University of Iowa, 
Iowa Center for Evaluation Research
College of Public Health

USA
Iowa, IW

Practice
Training
Teaching
Research
Services in design and conduct of evaluation procedures in ongoing University and state public health projects and programs  4 Faculty members
2 graduate students
Funding support from federal and state agencies, and private foundations
Research grants
University of Melbourne,
Centre for Programme Evaluation;
Faculty of Education

AUSTRALIA
Melbourne

Practice 
Research
Teaching
Training
Commissioned evaluations, survey research in education, health, welfare, training.
Developmental activities and workshops
6 Faculty/research fellows
5 research associates
admin staff
Revenue generated through contract work, fees for training activities and services Service provider relationship with all levels of government: scope ranging from local to national
University of Minnesota
Center for Applied Research & Educational Improvement ; College of Education & Human Devel
USA
Minneapolis, MN
Research
Practice
Teaching
Training
Independent evaluation and policy research of school district or community-based programs, state agency-funded programs, and other projects; Collaborative research projects in schools 10 principal investigators and research fellows; research assistants; 
staff
Service provider relationship: state and federal agencies and foundations Fees for training services
University of Nebraska
Center for At-Risk Children's Services; Spec Ed/Community Disorder Department

USA
Lincoln, NB

Research
Practice
Teaching
Training
Program evaluation, including survey research, needs assessment, data management, proposal writing services 5 Faculty members, administrative personnel, students, staff including data coordinators Service provider relationship: federal and state levels of government; 
community and school-based agencies.
University Of New Mexico
Health Evaluation and Research Office; Dept Family and Community Medicine
USA
Albuquerque, NM
Research
Practice
Training
Program evaluation, continuous improvement, research design, strategic planning, research methods analysis Unclear: Director, associate director, 
staff, 
students
Service provider relationship to: clinicians, coalitions, community health educators, federal and state agencies, foundations, not-for-profit agencies, policymakers, public health program developers, and researchers.
University of North Carolina, Chapel Hill:
Evaluation, Assessment & Policy Connections
School of Education
USA
Chapel Hill, NC
Practice
Training
Teaching
Program evaluation and technical assistance and development services in childcare, higher education; school-university partnerships; substance abuse prevention; community planning 6 Faculty members;
5 graduate students
Sponsorship: Federal, state and local agencies, and private foundations
Service provider relationship: some state and local government
University of Technology Sydney, Centre for Health Econ. Research & Evaluation
Faculties of Business and Nursing
AUSTRALIA
Sydney
Research
Practice
Teaching
Training
Economic evaluation, technology assessment, program evaluation in health, complex interventions; quality of life assessment; policy analysis economic forecasting 5 academic staff,
10 research associates,
3 post-doc fellows, 
6 research officers, 
6 administrative staff
Sponsorship: State government, health public agencies;
Unclear if service provider relationship;
Fees charged to students for courses and workshops;
University of Wisconsin Extension: Program Development and Evaluation,
Cooperative Extension
USA
Madison, WI
Practice
Training
Training and technical assistance to plan, implement and evaluate high quality extension educational programs. 5 Faculty members, associated staff Sponsorship: State government funding
Service provider relationship with state agencies.
University of York, Centre for Health Economics,
Dept of Economics and Related Studies and the Dept of Health Sciences
UK
Heslington, England
Research
Practice
Teaching
Training
Health economic policy analysis, evaluation and health technology assessment: primary car; 
addiction research resource allocation, outcomes research, econometrics
40 research staff including Faculty,
support staff
students
Service provider/client relationship with central and local public agencies and European Union, among others
Sponsorship: unclear
Fees for training
Vanderbilt University, Center for Evaluation Research and Methodology, Vanderbilt Institute for Public Policy Studies USA
Memphis, MS
Research
Practice
meta-analytic techniques for policy research; evaluation of programs in juvenile justice; school readiness; dissemination to policy makers and practitioners 2 Admin: director, research coordinator;
7 research associates and analysts
graduate student assistants
Sponsorship: Federal and state research grants; 
private foundation grant
No mention of fee for service provision relationships.
Vanderbilt University,
Center for Evaluation and Program Improvement,
Peabody College (ed., human development)
USA
Memphis, MS
Research
Practice
Program evaluation and program improvement in health and education, child, adolescent and family mental health services; contextualized feedback intervention theory 3 Faculty members, 
10 researchers,
research assistants, postdoctoral fellows, graduate student assistants
Service provide/client relationship with federal and state agencies
Sponsorship: federal and state agencies, private foundations, and private corporations
Western Michigan University,
The Evaluation Center:
Vice President of Res.
USA
Kalamazoo, MI
Research
Practice
Training
Teaching
Program evaluation and community development in higher ed; schools; personnel;
science education; social/youth
standard setting; state/regional educational services; testing
4 Faculty members, 
6 researchers; consultants, graduate students, and other Faculty members as associates
Sponsorship from a wide variety of national and regional associations and organizations
Service provider relationship: some government, most often community sector
a Teaching implies link to university degree programs; Training implies professional development services to clients; Research implies creation of academic knowledge which may or may not be directly related to evaluation; Practice implies delivery of evaluation services including consultation and project management.

The implementation of the model did not come about without its challenges, however, according to its Director, R. Taylor (personal communication, April 7, 2006). The GSRU ran a competition among a number of leading UK universities but few universities were interested in becoming an academic partner under the terms proposed. They wanted to run their own degree programs and could not understand the government's interest in partnership. Once the partner was identified, the major challenge was the degree accreditation system which was highly bureaucratic and took time to be approved by the University. The most significant obstacle was the establishment of a viable business model.

The GSRU paid a start up fee to the University partner and now charges a fee per student which covers off all costs. Even non-MSc students participate in the courses at a commercial rate. The first cohort in 2005 consisted of 18 students. To date feedback has been very positive. Students are quite satisfied with the program and Government departments are of the view that the degree offers good value for money and his helping raise skill levels. The program will undergo formal evaluation in the near term (R. Taylor, personal communication, April 7, 2006). We now continue our examination of government-university relationships by examining the concept of 'centre of excellence in evaluation' and how such centres have been and may be involved in ensuring quality in government level evaluation.

 

4. Survey of University-based Evaluation Centres of Excellence

Unlike training and educational programs in evaluation, there is very little appearing in the literature about the nature, roles and consequences of centres of excellence in evaluation. We sought to survey extant university centres both outside of Canada (in English speaking countries) and within her borders. It should be noted at the outset, that such units are most often called research centres or sometimes research groups or institutes. We sampled primarily through internet search engines, but also through bibliographic follow-up and telephone and email consultations and endeavoured to describe centres in terms of structural arrangements, functions and mission, sustainability, links to training and degree programs, and relationships with sponsors and clients (e.g., foundations, government). As with training programs our intention was to be comprehensive but we acknowledge that many centres that do extensive evaluative work are difficult to locate by virtue of the term 'evaluation' not appearing in their official name or mission statement. Nevertheless, in the sections to follow we attempt not only to describe extant centres but to develop a sense of the potential relationships with government either as service providers (contractors) or as recipients of government support or sponsorship. We begin with an examination centres located outside of Canada and conclude with a look at domestic centres of excellence with significant interest in evaluation.

 

4.1 University-based Centres of Excellence with an Interest in Evaluation located beyond Canadian Borders

4.1.1 Sample characteristics
In the end we located 21 centres of excellence with what we judged to be significant interests in evaluation. These are summarized in Table 5 and further details on each are located in Appendix B [3] . The majority (14 or 66%) are located in the US with 3 each in the UK and Australia and 1 in New Zealand. Twelve of the centres (57%) are located in faculties of education and human development (e.g., Harvard University, University of East Anglia, University of Melbourne), while 3 are located in Faculties of Medicine or Schools of Public Health (University of Aberdeen, University of Iowa, University of New Mexico), and 1 each in organizational behaviour/psychology (Claremont Graduate University), business studies (Massey University), and public policy (Vanderbilt University). A second centre located at Vanderbit University was in Education and Human Development. One centre (Royal Melbourne Institute of Technology) is not located within a Faculty or department but rather, reports directly to senior administration of the Institute. Three centres were interdisciplinary: University of Illinois Urbana-Champaign (psychology and education), University of Technology Sydney (business and nursing) and University of York (Economics and Health Sciences). It is interesting to note that 17 of the 21 centres (81%) had the term 'evaluation' explicitly represented in the centre name ('applied research' in one case).

4.1.2 Centre activities and supports
From mission statements and lists of activities we coded the principal activities of the centres into research (usually discipline specific but sometimes research on evaluation), practice (most often evaluation and related practices), teaching (formal links to degree programs offered at the university), and training (professional development in the form of non-credit workshops, seminars, institutes). Nine centres (e.g., Claremont Graduate University, University of Iowa) are engaged in all four types of activity, whereas most others are engaged in three of the four. A small number of universities are engaged in only two of these activities, usually being research and practice with no professional development or educational services offered.

As was the case with evaluation training programs, it was sometimes difficult to identify precisely how many faculty members and associated staffs were affiliated with the respective centres. Faculty complements of those with direct accountability for centre activities ranged quite enormously from about 2 to well over 10. Usually, centres had (sometimes extensive) lists of affiliated or associated members from various departments and faculties around campus. While most had identifiable support staff in the role of research coordination, project leadership, research assistance, and financial and administrative assistance, many also involved directly students in the some of these roles. In one of the columns in Table 5, we elaborated practice and consultancy activities in an effort to capture the nature of the evaluation-related business of the centres. Such activities could usually be categorized as one of three main types: consultation and advisory activities; evaluation and applied research delivery services (conducting the inquiry); and dissemination and follow up. In some instances, given the centre's mandate or mission, evaluation was integrated with other activities and responsibilities. For example, at the University of Illinois, Urbana-Champaign, evaluation was integrated with program development activities. In other locations, such as the University of East Anglia and the University of Minnesota, evaluation represented one choice on a menu of inquiry activities that included policy research, needs assessment or even action research. Consultation services are provided by many centres on such issues as funding proposal development, evaluation planning and design, and instrument design and validation. It might also include providing specific technical services such as computerized data scanning, or statistical analysis of data sets. 

In terms of dissemination activities, we observe that in some cases the centre acted as a knowledge brokerage with explicit goals of diffusing research and best practices not actually produced by centre personnel. But in many cases, however, such dissemination and follow up did actually relate to project work undertaken by centre staff.

In the final column in Table 5, we attempted to capture a sense of centre sustainability and where possible, to identify links to government. It would be safe to assume that virtually all centres would receive some sort of internal support from the university but that there would be expectations that the centres would be largely self-sustaining over time. We found it virtually impossible to identify from websites to what extent centres relied on internal support. We determined that significant internal support is provided in two cases: University of Melbourne (J. Owen, personal communication, April 7, 2006) and Western Michigan University (A. Gullickson, personal communication, April 11, 2006). In both cases, there is significant expectation the university support is augmented through the generation of external contract work and other revenue streams.

In many cases, we were able to ascertain that significant support was derived from external sponsorship, service provision (contract work), or fee for services such as workshop, conferences, short courses, and consultation. Sponsorships took one of two forms. First, external agencies (government, private foundations, private corporations) were sometimes identified as official sponsors of the centre, which we took to imply that they were supported by grants and contributions. Some centres actually got their start this way. For example, at UCLA the original Center for the Study of Evaluation (now the Center for Research on Evaluation, Standards, and Student Testing) was funded as a national research centre by the Department of Education. Other such centres were located at University of Wisconsin and the University of Pittsburgh and they all receive ongoing renewable funding (M. Alkin, personal communication, April 11, 2006). In other cases, sponsorship takes the form of grants obtained through competitive processes, usually from government sources or foundations. Federal, state and municipal governments were implicated as sponsoring agencies with municipal government being mentioned quite infrequently. At the University of Melbourne, state government contracts (and some federal and municipal) represent 80% of Centre revenues (J. Owen, personal communication, April 7, 2006). In other instances community agencies (often para governmental) served as clients for service provision. A similarly high proportion of government sourcing was reported with respect to Western Michigan University (A. Gullickson, personal communication, April 11, 2006).

4.1.3 Summary
In summary, we located a wide range of centres in five English- speaking countries around the globe. Many of the centres were located in faculties of education or human development but, health services and interdisciplinary centers were also noted to have a presence. Centres varied quite substantially in size and in the scope of their work. Most were involved in some combination of research; evaluation related practices including consultation, service delivery, and dissemination; and training or education. Most centres were dependent in some way shape or form on government (usually federal or state) for sponsorship, source of competitive grant funds, or contracted project work. We observed that private foundations often provided support as well. Centre business often included disciplinary research (e.g., child welfare, public health) in addition to evaluation-related services. In some instances we observed formal links to degree programs but sometimes center activities did not involve education or training. We now turn to an examination of domestic Canadian centres of excellence by way of examining similarities and differences with those located in the international context.

 

4.2 Canadian University-based Centres of Excellence with an Interest in Evaluation

4.2.1 Sample characteristics
Whereas over 80% of the centres comprising the international sample included 'evaluation' in the centre name, such was the case in only 5 (62%) of our final sample of 8 Canadian centres. At least partially due to this reason, centres with a significant interest in evaluation were comparatively more difficult to locate. Ultimately, as shown in Table 6, we located 1 centre in the Maritime Provinces (University of New Brunswick), 5 in Ontario (Carleton University, Queen's University, University of Ottawa, University of Toronto, University of Waterloo) and 2 in the west (University of Calgary, University of Saskatchewan/University of Regina). The center in Saskatchewan is actually a jointly managed organization comprising two universities and a health foundation. Three of the centres are either not affiliated with specific faculties or departments or that information is not clear on the website (University of New Brunswick, University of Calgary, University of Saskatchewan/University of Regina). The others are disciplinarily located in the public management (Carleton University), education (Ontario Institute for Studies in Education/University of Toronto), social sciences (University of Ottawa), or health sciences (University of Waterloo). As are the unaffiliated centres, the centre at Queen's University is interdisciplinary (Faculties of Education and Health Sciences).

We located four other centres but decided that their interest in evaluation (as understood for the purposes of this paper) was marginal although they are very much involved in important work with great potential to influence government policy. These were:

  • University of Manitoba, Faculty of Medicine, Manitoba Centre for Health Policy
    http://www.umanitoba.ca/centres/mchp/: The centre conducts health research that feeds into provincial health policy and currently receives 2/3s of its funding from Manitoba provincial government.
  • McMaster University, Centre for the Evaluation of Medicines
    http://www.thecem.net/index.html: This centre provides expertise in clinical pharmacology and toxicology, economics, health policies, behavioural sciences, information technology, research design and biostatistics. Specializations include: clinical pharmacology studies; health appraisal; technology assessment; health services research of traditional and complementary therapies; adverse drug reactions and toxicology data; health policy interventions on patient outcome and health care costs.
  • University of Alberta, Department of Medicine, Centre for Health Evidence ;
    http://www.cche.net/default.asp This multi-disciplinary initiative brings universities, health organizations, and professional associations together to support learning, teaching, and practice of evidence-based health care.
  • Institute for Social Research, York University,  
    http://www.isr.yorku.ca/home.html The Institute for Social Research provides five types of services: survey research, focus groups, statistical consulting, data archives, and data entry/data management.

In contrast to the international sample, none of the Canadian centers demonstrated a scope of activity all of the identified areas - research, practice, teaching, training - although most appeared to focus on three of the four. In all cases, research was a main activity of the centre, and this we took to imply disciplinary research (policy, public health) as opposed to research on evaluation. Practice activities related to evaluation were similar in Canadian centres to those from the international sample. Specifically, several centres provided program evaluation services and consultation and/or dissemination and follow up services designed to foster evidence-based practice. In some cases, such as University of Calgary and Carleton University, evaluation services were somewhat incidental to the more mainstream policy research activities.

Formal links to university degree programs were observable in only one of the centres (University of Saskatchewan/University of Regina,) whereas several centres (e.g., University of Ottawa, University of Waterloo) identified evaluation training and capacity building as central activities. It is interesting to note that only two of the centres (University of New Brunswick, University of Ottawa) make explicit reference to the involvement of students in centre business. Although the number of faculty dedicated to the centre was difficult to determine in some instances, most identified 4 or 5 faculty in addition to research fellows, associates or faculty affiliates. It was not possible to determine staffing parameters for Carleton University or OISE/UT from their respective websites. Perhaps as no surprise given the Canadian population, we did not observe excessively large centers as was the case in the US and the UK.

Table 6: Centres of Excellence in Evaluation and Related Fields in Canada

Name Location Mission a   Practice /Consultancy Human Resources Sustainability / 
Link to Government
Carleton University, Centre for Policy and Program Assessment, School of Public Management Ottawa, ON Practice
Research
applied research in, numerous public policy fields and program areas at the federal, provincial municip. and international levels of government. Unclear: several Faculty from different departments and disciplines Service provider relationship with federal government
Sponsorship: unclear (federal government research grants)
Ontario Inst for Studies in Education/UT Centre for the Advancement of Measurement, Evaluation, Research & Assessment Toronto, ON Practice
Research
Training
Collaboration on R & D;
program evaluation design; instrument development and validation; data collection and analysis; report writing
Workshops, symposia and seminars on methods issues 
Unclear;
1 Faculty person
Revenue generated through contract work
Service provider relationship with provincial government
Queens University, Social Program Evaluation Group
Faculty of Education, Faculty of Health Sciences
Kingston, ON Practice
Research
Basic, applied and policy research; program evaluation and monitoring; dissemination activities with partner agencies  1 Director, 2 Faculty, 4 project managers, 1 research associate, 2 support staff. Service provider relationship: contracts with federal and provincial government
Sponsorship: federal and provincial research grants. 
University of Calgary, 
Institute for Advanced Policy Research,
Unaffiliated
Calgary, AB Research
Training
Practice
No mention of evaluation practice. Focus on policy research. Policy briefs and technical report dissemination. Cities, disabled, well being, climate change. 1 Director
4 Faculty
28 affiliated Faculty and staff
Funding and sustainability are unclear
Sponsorship through competitive research funding.
University of New Brunswick
Canadian Research Institute for Social Policy,
Unaffiliated
Fredrickton, NB Research
Practice
Conducting detailed evaluations of local, national, and international policy initiatives, and by analyzing large complex data bases 2 administrators (director, associate)
4 research fellows, 
2 research associates, 
2 staff and 6 students
Funding and sustainability are unclear
Sponsorship through competitive research funding.
University of Ottawa
Centre for Research on Community Services:
Faculty of Social Sciences
Ottawa, ON Research
Practice 
Training
Social research studies; community program devel through research and training; needs assessment; program evaluation; survey design and analysis 2 principal Faculty, 
Faculty, 
9 senior researchers, Research coordinator, student assistants, 
admin staff
Revenue generated through contract work, fees for training activities and services; research grants
Service provider relationship: some government, most often community sector
University of Sask & Regina
Saskatchewan Population Health And Evaluation Research Unit; 
University affiliation unclear

Saskatoon, Regina, Prince Albert, SK

Research
Practice
Teaching
Research focus on Aboriginal northern and rural health, children's health, policy and governance and health. No evaluation services identified 1 Administrator: director
6 Faculty
6 support staff
Partnership among two universities and Saskatchewan Health Research Foundation
Funded research.
University of Waterloo,
Centre for Behavioral Research and Program Evaluation
Faculty of Applied Health Services

Waterloo, ON

Research
Practice
Training
Evaluation planning; data collection tools and protocols; data analysis and interpretation; knowledge synthesis and translation; capacity building and training 3 admin (director, 
2 assistant directors)
3 scientists, 
7 admin support staff, 
12 evaluation and research staff.
Sponsored by Canadian Cancer Society. Other funding unclear: competitive research funding likely.
a Teaching implies link to university degree programs; Training implies professional development services to clients; Research implies creation of academic knowledge which may or may not be directly related to evaluation; Practice implies delivery of evaluation services including consultation and project management.

Finally, funding and modes of sustainability were generally difficult to determine but it would be safe to say that at least a portion of centre budgets would come from the university while other means of revenue generation would come through sponsorship and securing competitive grants as well as through contract work. There seems to be ample evidence to show that the centre sustainability depends in part on government sponsorship or contract services to government. In the case of University of Waterloo and University of Saskatchewan/University of Regina, formal partnerships were established with supporting non-governmental agencies and foundations.

4.2.2 Summary
To summarize, compared to the international sample, Canadian centres of excellence with significant interest in evaluation-related activities, appear to be somewhat more homogeneous in size and less prevalent in faculties of education. We observed a tendency for interdisciplinary centers to exist and the centres do not appear to be affiliated with a particular faculty or department. Centres that participate in evaluation-related activities were difficult to locate by virtue of evaluation not being represented in the centre name. Nevertheless, there is substantial involvement of university-based centres in Canada in evaluation activities, either in consultation, service delivery, or training. There is also a good deal of interest in fostering evidence-based practice in the respective field of practice, in some cases through disseminating policy research or brokering research done elsewhere. Finally, it seems clear that centres are dependent to a significant degree on funds generated through their relationship with government, either as a recipient of sponsorship, grant recipient or as a contractor to government at provincial and/or federal levels.

 

5. Conclusions and Implications

The foregoing literature review and survey of university-based evaluation training options and the existence, function and sustainability of university-based centres of excellence with an interest in evaluation provides a comprehensive platform from which to consider potential roles for government in fostering evaluation quality assurance. Given the deliberations and the evidence concerning university-based interests in evaluation in Canada that we uncovered, we are persuaded that it would be premature at this time to move to an individual-level certification model. Although in Canada and elsewhere significant recent progress has been made in developing core competencies for evaluators, diversity in the field of practice is substantial and it represents a serious mitigating factor against the implementation of a licensure approach that would restrict entry into the field by virtue of tests of minimum levels of knowledge and skill. Further, we have seen that graduate-level degree programs in evaluation are just not available in Canada unlike the situation in other jurisdictions. There is, however, substantial graduate level instruction concerning the evaluation function occurring in Canadian universities and we found some indication that graduate certificate programs are becoming a realistic advanced-level option to existing professional development activities that result in a certificate of participation, rather than achievement.

Our primary conclusion from this analysis is that a system of credentialing which would acknowledge a set of courses or other experiences a person must go through to be recognized would be the most prudent and realistic route to meeting current demands in the Canadian context for quality assurance in evaluation. Such a system could form the basis of a more elaborate and stringent certification system in time, should consensus of the definition and bounded competencies that evaluators should possess. It is on this central plank that we now turn to considerations for the role of government in fostering evaluation quality assurance in Canada. We address such issues under the banners of training, centres of excellence, other implications for universities and links with the professional society, the CES.

5.1 Training and Education

5.1.1 Graduate certificate programs in evaluation
The development of pilot projects of graduate-level university certificate programs in program evaluation represents a reasonable and potential powerful step for government to take in fostering its quality assurance agenda. These would be master's level programs that would include 5 or 6, 3-credit courses to be done on a part- time or full-time basis. The Ontario Council for Graduate Studies has become quite open to the concept in recent years, as Ontario universities have benefited from the development and implementation of graduate certificates in a variety of applied fields and domains of inquiry. We note that the concept of graduate certificate program is becoming commonplace in other jurisdictions in Canada, as it is globally. The focus would be on preparing qualified and competent persons to assume evaluation roles. Programs should include solid grounding in methods and practice of evaluation, evaluation theory and models and experiential learning opportunities through practica or internship placements.

A small number of these programs currently exist in Canada. It might be possible to partner with these programs to offer specialized versions of the certificate program that would be tailored to the needs for evaluation in the federal government. As well, it may prove beneficial to assist universities to develop distance education approaches to delivering these programs so that they are made available to federal sector employees across the country.

Other possibilities would be to negotiate pilot opportunities with other promising sites in Canada, such as university faculties or departments that currently offer multiple evaluation courses on a regular basis. Support in the form of guaranteeing a certain number of federal government placements (i.e., government personnel to be retrained for evaluation) over coming years would be useful to help establish and develop the programs within the university structure.

The advantage of federal support for the development of graduate certificate programs in evaluation might be realized in the form of federal-level credentialing of evaluators. That is, the federal government could move ahead and require its program evaluators or contractor to have completed a graduate-level certificate program in evaluation (presumably one that would have participated in or have been modelled on those involved in the pilot initiative). The credential then becomes the certificate which graduates of the program receive. This route assumes reasonable similarity between the different programs across the country, which there should be if coordinated via a pilot project. (Another possibility would be to work with CES to develop a registry of credentialed evaluators - see discussion below).

5.1.2 Develop a graduate degree program in program evaluation in partnership with a university
A somewhat more ambitious option would be to collaborate with one or more universities to develop a graduate degree that is specifically tailored to meeting government evaluation training and certification needs. Such a program would provide a significant challenge to develop and install and would implicate ongoing commitment to running the program by government. A core curriculum for such a program could be developed on the basis of what is currently known about evaluator competencies juxtaposed to identified, and perhaps somewhat unique, government needs with regard to the evaluation function (e.g., fit with expenditure management and accountability framework). The program could be offered on a part-time basis to select public servants on a pre-service basis, or perhaps on an in-service basis in the short run. The degree would be recognized on a university transcript and therefore transferable anywhere, which would be likely to be highly attractive to a good many public servants. Another advantage of such a program is that curriculum would be tailored to government needs and therefore highly relevant, while at the same time quality would be assured by mandatory compliance with provincial regulations for graduate instruction. In order to accomplish the latter, their may be implications for involvement of university faculty in aspects of courses that are given by members of the public service. (Recall, for example, that provisions for assignment development and marking to be the purview of the Institute of Education are associated with the London program - see section 3.3 above.) Another consideration would be geography. Would the program be made available to public servants across the country and if so, on what basis? Finally, the program would necessarily be given in both official languages which would carry resource implications.

5.1.3 Workshops, short courses, summer institutes and other learning experiences
Several possibilities exist here to continue to provide more basic level training and exposure to evaluation principles and practices. First, CEE should continue to develop and offer workshops and learning events associated with a CEE evaluation learning strategy. These events are highly relevant to evaluation in government and may serve to augment more advanced training such as degree programs or graduate certificate programs. Similarly, it would be beneficial to encourage basic-level training participation in CES Essential Skills Series and intermediate short courses. It would be prudent to evaluate the curricula of these courses against contemporary government exigencies in order to ensure that there would at least partly meeting quality assurance needs. Another option might include developing partnerships with universities to offer summer institutes that could be theme-based or more general to evaluation capacity building. Universities have considerably more flexibility to offer such courses because they are not governed by central accrediting agencies and ultimately are not included on the University transcript. Such institutes are quite popular in other jurisdictions and offer the opportunity to bring in high profile guest speakers from within the evaluation community.

 

5.2 Centres of Excellence

5.2.1 Support for development of university-based centres of excellence in evaluation
Our analysis shows the multiple dimensions of value that university-based centres of excellence can add to the evaluation quality assurance agenda. Yet in Canada, the existence of centres with wide scope with regard to the evaluation function is somewhat muted as compared to other jurisdictions. Only some existing centres have formal ties to graduate degree or certificate programs, and since there are really no evaluation degree programs in Canada, they represent one avenue to training highly qualified evaluation personnel that is largely underdeveloped. Some existing centres of excellence do include training and evaluation capacity building as part of their core activities. Such functions could be invaluable to public servants needing to develop specific knowledge and skill sets on an in-service basis. Centres of excellence also serve a consultative function. It would be beneficial for government departments to cultivate such relationships and to take advantages of the function for advisory and peer review services. Centres of excellence also carry out contract work and provide bona fide evaluation services. Government investment in stimulating centre development would be well spent to the extent that such centres could provide alternatives to the usual firms located on standing offers for service provision. In other jurisdictions, particularly in the US, evaluation centres have the infrastructure to handle very large scale evaluations of national programs. There is no reason why this could not be the case in Canada. It is true that individual professors are unable to drop their myriad of commitments to pick up demanding contracts that are on tight timelines, yet appropriately resourced centres would comprise research associates, coordinators, post doctoral fellows, and students, all of whom would be in position to provide valued expertise to evaluations of major program or policy initiatives.

How could government help bring this about? For one, through internal restructuring within government, contract work could be made more readily available or accessible to universities and university centres in particular. In the interest of developing a core of highly qualified personnel in evaluation, another consideration would be to establish funding post-doctoral experiences working with such organizations. Fostering research on evaluation, another potentially strong interest of centres of excellence, would provide another option for consideration. Despite their different purposes, research interests can often be piggybacked on evaluation activities. A caveat would be that universities would require at least shared ownership of intellectual property, but stimulating research on evaluation could be highly advantageous in the long run. Good research attracts interest and in and of itself may serve a capacity building function with regard to the appreciation of the power and potential of evaluation as a management function. In addition to post-doctoral opportunities, the development of a small number of research chairs dedicated to program evaluation and associated with certificate programs could be considered as a means of advancing the ability of universities to contribute to the development of program evaluation capacity across the country.

5.2.2 Continue/expand academic liaison
Quality assurance bodies such as advisory committees for ongoing evaluation planning and integration with strategic plans, peer-review of evaluation frameworks and evaluation reports, and meta-evaluations of clusters of evaluation reports are all valued contributions that can be made by academics regardless of whether they are affiliated with centers of excellence. This is current practice in many federal government departments and agencies but it is far from widespread. It would be prudent to encourage government departmental and agency evaluators and members of the decision and policy community to cultivate relationships with academics in such advisory capacities. We observe that many evaluation courses are offered at universities across the country and that it is likely that a good deal of evaluation expertise exists in universities despite the paucity of official evaluation degree and certificate programs.

 

5.3 Other Involvement with Universities

5.3.1 Exchanges between government sector and academe
Longstanding support in the organizational change literature exists for the prospect of developing organizational knowledge and learning through attracting external persons such as academics to work in the government sector for short term assignments. In the present case these might take the form of one year secondments, sabbatical placements, or short term replacements for persons on temporary leave. Datta (2003) points out that some of the more influential figures in developing evaluation as a domain of inquiry have been academics who have worked in the government sector with significant evaluation responsibility and influence.

5.3.2 Support for student development
Students represent the next generation of evaluators and efforts to stimulate the development of their knowledge and skill in evaluation would represent a class of strategies that would be likely to pay off. Many evaluation centres of excellence routinely engage students in evaluation contract work, which is typically extraordinarily beneficial to all concerned. In addition to helping students to secure financial sustainability, knowledge and skill developed in the practical milieu are likely to be robust. Such opportunities need not be limited to contract work in the centre of excellence. Summer placement programs and other modes of involving students in work placements, such as government internship sites, could be highly beneficial. Currently cooperative education programs exist at the graduate level at only a small number of universities and we can say with some certainty, none with evaluation as a focus. Yet this mode of learning carries with it a long tradition of support in terms of attendant benefits to both students and the organizations which take them on. In addition to such direct support, continued sponsorship of worthwhile competitions such as CES paper contests and case competitions would be likely to pay off for government as well.

 

5.4 Support for Professional Society

5.4.1 Support CES to develop credentialing system
In section 5.1.1 above we argued that an easy and logical feature of a pilot program to set up graduate certificate programs at universities across the country would be for the government to institute a defacto credentialing system. Public servants and external persons completing the graduate certificate would be considered competitively qualified for upcoming positions. Yet credentialing systems can and should be more sophisticated than that in order to minimize the production of false negatives (persons not credentialed who should be) and false positives (persons credentialed who should not be). In addition to the certificate program, there may be other experiences such as disciplinary graduate degrees with specialization in evaluation, evaluation practical experiences, contributions to the advancement of evaluation theory and practice, that might ultimately help to address the problem. Yet such systems would imply the establishment of procedures for adjudicating applications and for maintaining a registry of qualified persons. Such persons could be working directly within government or external to it as private consultants.

Developing such a credentialing system, as we suggested above, might be viewed as an incremental strategy toward eventual certification. And given the professional interests at stake it would be logical for the professional association to take the lead in installing such a system. Of course, the development of a credentialing system would incur substantial start up costs and maintenance costs for which subsidization might be required. For example, government might consider support for a conference or some process to agree to a common set of standards for credentialing, accreditation of programs, and the like. Or a formalized partnership in a credentialing system might be an option worth considering. Once installed full-time staff would be required to maintain a registry of credentialed evaluators, and that might represent substantial member dues increments. With a credentialing system in place, and a mechanism to equate other experiences with the demands of a graduate certificate, TBS might consider implications for hiring practices within the federal sector and for contracting out to private service providers.

 

5.5 Closing Remark

This brings to a close our thoughts about possible roles for government in assuring quality in evaluation. The prospects for the development and cultivation of government-university relationships are many and varied. Both sectors, it seems, stand to benefit quite enormously from heightened assurances of evaluation quality and ultimately the full integration of evaluation into the management function. We would hope that this discussion paper in some significant way lays the groundwork for further concrete dialogue, deliberation and, ultimately, action toward these ends.

 

Footnotes

  1. It should be noted that TBS evaluation competencies were based on a competency profile study of the federal evaluation community.
    back
  2. Appendix A available online at http://www.tbs-sct.gc.ca/cee/tools-outils/qae-aqe/annexa-eng.asp  
    back
  3. Appendix B available online at http://www.tbs-sct.gc.ca/cee/tools-outils/qae-aqe/annexb-eng.asp
    back

 

References

American Evaluation Association (2004) American Evaluation Association Response to U.S. Department of Education notice of proposed priority, Federal Register RIN 1890-ZA00 , November 2003 "Scientifically based evaluation methods" Retrieved April 19, 2004 from http://eval.org/doestatement.htm

Amara, N., Ouimet, M., & Landry, R. (2004). New evidence on instrumental, conceptual and symbolic utilization of university research in government agencies. Science Communication , 26, 75-106

Alkin, M. C., Kosecoff, J., Fitz-Gibbon, C., & Seligman, R. (1974). Evaluation and decision making: The Title VII experience . Los Angeles, CA: Centre for the Study of Evaluation.

Altschuld, J. W. (2005). Certification, credentialing, Licensure, competencies, and the like: Issues confronting the field of evaluation. Canadian Journal of Program Evaluation , 20(2), 157-168.

Altschuld, J. W. (1999). Certification of evaluators: Highlights from a report submitted to the Board of Directors of the American Evaluation Association. American Journal of Evaluation , 20, 481-493.

Altschuld, J. W. (1999). A case for a voluntary system for credentialing evaluators. American Journal of Evaluation , 20(3), 507-517.

Altschuld, J. W., Engle, M., Cullen, C., Kim, I., & Macce, B. R. (1994). The 1994 directory of evaluation training programs. In J. W. Altschuld & M. Engle (Eds.), New Directions in Program Evaluation: The preparation of professional evaluators; issues perspectives, and programs , No 62 (pp. 71-94). San Francisco: Jossey-Bass.

Aucoin, P. (2005, April). Decision making in government: The role of program evaluation. Ottawa: Centre of Excellence in Evaluation, Treasury Board of Canada, Secretariat.

Auditor General of Canada. (1997, Oct.). Moving toward managing for results. Report to Parliament, Chapter 11. Ottawa:: Author.

Bickman, L. (1999). AEA, bold or timid? American Journal of Evaluation , 20, 519-520.

Breen, G., & Associates. (2005, May). Interviews with deputy ministers regarding the evaluation function. Ottawa: Centre of Excellence in Evaluation, Treasury Board of Canada, Secretariat.

Borys, S., Gauthier, B., Kishchuk, N., & Roy, S. (2005, November). Survey of evaluation practice and issues in Canada . Paper presented at the Joint Canadian Evaluation Society/American Evaluation Association Conference, Toronto.

Canadian Evaluation Society (n.d.-a). Essential skills series . http://evaluationcanada.ca   retrieved January 20, 2006.

Canadian Evaluation Society (n.d.-b). CES guidelines for ethical conduct . http://evaluationcanada.ca   retrieved January 20, 2006.

Centre of Excellence for Evaluation. (2004). Review of the quality of evaluations across departments and agencies . Ottawa: Author, Treasury Board of Canada, Secretariat.

Cousins, J. B. (2005). Interview with Joe Hudson founding editor of the Canadian Journal of Program Evaluation. Canadian Journal of Program Evaluation , 20(3), 199-221.

Cousins, J. B., Goh, S., Clark, S., & Lee, L. (2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation , 19(2), 99-141.

Cousins, J. B., Goh, S., Aubry, T., Elliott, C., Lahey, R., & Mongtague, S. (2006, January). What makes evaluation useful? A concept mapping study . Presentation made at a session of the Performance and Planning Exchange, Ottawa.

Datta, L.-E. (2003). The evaluation profession and the government. In T. Kellaghan & D. L. Stufflebeam (Eds.), International Handbook of Educational Evaluation (pp. 345-360). Boston: Kluwer.

Engle, M., & Altschuld, J. W. (2003). An update on university-based evaluation training. Evaluation Exchange , 9(4), 13.

Engle, M., Altschuld, J. W., & Kim, Y. C. (in press). 2002 survey of evaluation preparation programs in universities: An update of the 1992 American Evaluation Association-sponsored study. American Journal of Evaluation .

Government of Canada. (2000). Results for Canadians. Ottawa: Author. 

Government of Canada (2006). Canada's new government. Federal accountability action plan. Turning a new leaf . Ottawa: author. http://www.faa-lfi.gc.ca/docs/ap-pa/ap-pa00_e.asp retrieved April 15, 2006.

Grasso, P. (2003). What makes an evaluation useful? Reflections from experience in large organizations. American Journal of Evaluation , 24, 507-514.

Gussman, T. K. (2005, May). Improving the professionalism of evaluation . Ottawa: Centre for Excellence in Evaluation, Treasury Board Secretariat of Canada.

Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards . Thousand Oaks: Sage.

Jones, S. C., & Worthen, B. R. (1999). AEA members' opinions concerning evaluator certification. American Journal of Evaluation , 20, 495-506.

King, J. A., Stevahn, L., Ghere, G., & Minnema, J. (2001). Toward a taxonomy of essential evaluation competencies. American Journal of Evaluation , 22, 229-247.

Kingsbury, N., & Hedrick, T. E. (1994). Evaluator training in a government setting. In J. W. Altschuld & M. Engle (Eds.), New Directions in Program Evaluation: The preparation of professional evaluators; issues perspectives, and programs , No 62 (pp. 61-70). San Francisco: Jossey-Bass.

Landry, R., Amara, N., & Lamari, M. (2001). Utilization of social sciences research knowledge in Canada . Research Policy, 30, 333-349.

Leviton, L. C. (2003). Evaluation use: Advances, challenges and applications. American Journal of Evaluation , 24, 525-535.

Long, B., & Kishchuk, N. (1997). Professional certification: A report for the National Council of the Canadian Evaluation Society on the experience of other organizations . Ottawa: Canadian Evaluation Society.

Love, A. (1994). Should evaluators be certified? In J. W. Altschuld & M. Engle (Eds.), New Directions in Program Evaluation: The preparation of professional evaluators; issues perspectives, and programs , No 62 (pp. 29-40). San Francisco: Jossey Bass.

Mathison, S., & (Ed.). (2005). Encyclopedia of evaluation . Thousand Oaks: Sage.

May, R. M., Fleisher, M., Schreier, C. J., & Cox, G. B. (1986). Directory of evaluation training programs. In B. G. Davis (Ed.), Teaching evaluation across the disciplines. New Directions in Evaluation , No. 29 (pp. 71-98). San Francisco: Jossey Bass.

McGuire, M., & Zorzi, R. (2005). Evaluator competencies and professional development. Canadian Journal of Program Evaluation , 20(2), 73-99.

Morris, M. M. (1994). The role of single evaluation courses in evaluation training. In J. W. Altschuld & M. Engle (Eds.), New Directions in Program Evaluation: The preparation of professional evaluators; issues perspectives, and programs, No 62 (pp. 51-59). San Francisco: Jossey Bass.

Morris, M. M. (2003). Ethical considerations in evaluation. In T. Kellaghan & D. L. Stufflebeam (Eds.), International Handbook of Educational Evaluation (pp. 303-328). Boston: Kluwer.

Muller-Clemm, W. J., & Barnes, M. P. (1997). A historical perspective on federal program evaluation in Canada. Canadian Journal of Program Evaluation , 12(1), 47-70.

Nagao, M., Kuji-Shikatani, K., & Love, A. (2005). Preparing school evaluators: Hiroshima pilot test of the Japan Evaluation Society's accreditation project. Canadian Journal of Program Evaluation , 20(2), 125-155.

Office of the Comptroller General. (1981a). Guide on the program evaluation function . Ottawa: Minister of Supply and Services.

Office of the Comptroller General. (1981b). Principles for the evaluation of programs by federal departments and agencies . Ottawa: Minister of Supply and Services.

Patton, M. Q., Grimes, P. S., Guthrie, K. M., Brennan, N. J., French, B. D., & Blyth, D. A. (1977). In search of impact: An analysis of the utilization of federal health evaluation research. In C. H. Weiss (Ed.), Using Social Research in Public Policy Making (pp. 141-163): D. C. Heath and Company.

Perrin, B. (2005). How can the information about the competencies required for evaluation be useful. Canadian Journal of Program Evaluation , 20(2), 169-188.

Russon, C., Russon, G., & (Eds.). (2005). International perspectives on evaluation standards. New Directions in Evaluation , No. 104. San Francisco: Jossey-Bass.

Segsworth, R. V. (2005). Program evaluation in the Government of Canada; Plus ça change... Canadian Journal of Program Evaluation, 20(3), 173-195.

Shadish, W., Newman, D., Scheirer, M. A., & Wye, C. (Eds.). (1995). Guiding principles for evaluators. New Directions in Evaluation . No.66. San Francisco: Jossey-Bass.

Smith, M. F. (2003). The future of the evaluation profession. In T. Kellaghan & D. L. Stufflebeam (Eds.), International Handbook of Educational Evaluation (pp. 373-386). Boston: Kluwer.

Smith, M. F. (1999). Should AEA begin a process of restricting membership in the professional of evaluation? American Journal of Evaluation , 20, 521-531.

Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005a). Establishing essential competencies for program evaluators. American Journal of Evaluation , 26, 43-59.

Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005b). Evaluator competencies in university-based training programs. Canadian Journal of Program Evaluation , 20(2), 101-123.

Stufflebeam, D. L. (2001). Interdisciplinary Ph.D. programming in evaluation. American Journal of Evaluation , 22, 445-455.

Scriven, M. (2003, November 12). Comments to feds re: proof of causation. Message posted to EVALTALK [Msg. 41]. http://bama.ua.edu/archives/evaltalk.html

Treasury Board of Canada, Secretariat. (1977). Evaluation of programs by departments and agencies . (Treasury Board Circular 1977-47) Ottawa: Author.

Treasury Board of Canada, Secretariat. (2001). Evaluation policy . Ottawa: Author. http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/tbm_161   retrieved January 20, 2006.

Treasury Board of Canada, Secretariat, (2006a). Policy on internal audit . Ottawa: Author. http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12340   retrieved April 15, 2006.

Treasury Board of Canada, Secretariat, (2006b). Guidelines on the Responsibility of Chief Audit Executive . Ottawa: Author. http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12352 retrieved April 15, 2006. 

United States Department of Education (2003). Scientifically based evaluation methods . (Federal Register RIN 1890-ZA00, November 4, 2003, vol. 68, no. 213, pp. 62445-62447). Washington, DC: U.S. Government Printing Office.

Worthen, B. R. (1999). Critical challenges confronting certification of evaluators. American Journal of Evaluation , 20, 533-555.

Worthen, B. R. (2003). How can we call evaluation a profession if there are no qualifications for practice? In T. Kellaghan & D. L. Stufflebeam (Eds.), International Handbook of Educational Evaluation (pp. 329-344). Boston: Kluwer.

Weiss, C. H., & Bucuvalas, M. J. (1980). Social science research and decision making . New York: Columbia University Press.

Zorzi, R., Perrin, B., McGuire, M., Long, B., & Lee, L. (2002). Defining the benefits, outputs, and knowledge elements of program evaluation. Canadian Journal of Program Evaluation , 17(3), 143-150.

Zorzi, R., McGuire, M. & Perrin, B. (2002). Canadian Evaluation Society project in support of advocacy and professional development: Evaluation benefits, outputs, and knowledge elements . Retrieved Jan. 20, 2006 from http://consultation.evaluationcanada.ca/pdf/ZorziCESReport.pdf

Date modified: