Treasury Board of Canada Secretariat
Symbol of the Government of Canada

ARCHIVED - Linkages between Audit and Evaluation in Canadian Federal Departments (TBS Paper) - September 30, 1993

Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.




TBS Paper 
Date: 09-30-93 

Subject: Linkages between Audit and Evaluation in Canadian Federal Departments

Government Review Division
Administrative Policy Branch
Treasury Board Secretariat

by Ian Hurd, BA (Carleton), PhD. candidate (Yale)





Table of Contents

Introduction

Forms of Linkage

Impacts of Linkage

Conclusions

Individuals Interviewed for the Study

References

 

 




Introduction

This paper will document the relationship between audit and evaluation in departments of the Canadian federal government and draw some lessons from the recent experiences of departments in studies involving both audit and evaluation.  It is based on interviews with fifteen individuals from departments' audit and evaluation units, ranging from officers of audit or evaluation to Assistant Deputy Ministers, an investigation of about ten other departments, and on a brief review of the current literature.

The first section will examine the theoretical issues behind linking audit and evaluation, including the forms of linkage existing in some Canadian federal departments.  The second part will examine the effects of these links on projects, clients, reviewers themselves, and the review groups as organizations.  To begin, however, we need a brief introduction to the language of audit-evaluation issues.

Discussion in the audit-evaluation field has in the past suffered from an inconsistent use of terms; a word about the terminology used in this paper therefore seems prudent.  Internal audit is defined in the Treasury Board Manual on Evaluation and Audit as the "assessment... against suitable criteria [of the].... collection of interrelated systems and practices adopted by management to achieve established objectives," in order that the deputy head and others may improve the cost-effective and accountable management of operations.  The same document defines program evaluation as an empirical "analysis of the performance of the program" so that the deputy head and others may reconsider "the future direction and resourcing of the program, including its design, delivery, and service levels."

Linkage herein refers to a condition in which audit and evaluation share organizational or functional space.  Organizational linkage describes a situation in which audit and evaluation groups are connected on a department's organization chart; this can range from both groups reporting to the same ADM, to complete fusion into a single group.  Functional linkage, by contrast, refers to an occasion or a system in which a review study deals with both audit and evaluation issues or in which both auditors and evaluators participate.  Any working relationship between audit and evaluation falls under the term functional linkage, but most interesting to this study are instances of the two functions being in formal and regular contact.

The term "review" is used here to refer to the audit and evaluation functions in the aggregate, and is mainly a stylistic device to spare the reader from tiring of the "audit and evaluation" construction.  Thus, the "review group" is that unit of a department responsible for audit and evaluation.  When capitalized, "Review" and "Reviewer" refer to the single discipline and its practitioner created in some departments by the fusion of audit and evaluation.

The present study was complicated by the June 1993 reorganization of government, which resulted in the abolition of several of the departments contributing to the project.  In numerous cases, departments with very different audit-evaluation relations were merged, and it is as yet unknown how the review groups for the new departments will be arranged.  This makes it difficult to cite the absolute number or proportion of departments choosing one form or another or to refer to groups by name, but the substance of their experiences and comments remains valid.

 




Forms of Linkage

History, Trends, and Theory

Audit and evaluation developed quite separately, at different times, and out of different disciplines. Audit has its history in accounting and financial management, and adopted their traditional independence from program management. It gets much of its strength from the fact that it has a largely agreed upon set of standards (IIA and federal). It delivers a range of products, from comprehensive to compliance audits, dealing with different aspects of an organization, but in all forms it takes as its starting point the management and operations of the organization, and moves outward to the organization's activities and products.

Evaluation, with its more recent heritage in the 1960s, arose in response to a perceived need in the United States for an understanding of the actual effects of military "Great Society" projects. It uses (largely social-) scientific methods, providing empirical evidence of the impacts and effects of public policies, intended to be used as inputs to decision making. Evaluation begins at the impacts and outputs of the organization and may move inward to examine the operational reasons for these results. Evaluation has developed its own set of standards (AEA and federal) for measuring performance, but it maintains a traditional connection with program managers and planners, valuing objectivity over independence in practice.

Reflecting the origins of each function, audit in the federal government has traditionally been attached to the financial management area and evaluation to policy development. While it is generally acknowledged that audit and evaluation form a continuum of organizational information services with overlap in areas such as efficiency of operations and cost effectiveness, they were traditionally kept separate, with evaluation concerned with analysis of policy and outputs, and auditing with internal financial controls and management systems.

These traditions have been changing in the last decade in many jurisdictions and in both private and public organizations. There appear to be two reasons for this in the public sector, both related to restraint. First, restraint in the audit and evaluation sectors means a new pressure on the functions to show their worth by meeting client demands for products. This requires flexibility in defining the list of available products, and a willingness to take on tasks which do not fit the traditional disciplinary boundaries. It appears the demand among clients is changing in the direction of the products providing information on operations, with the objective of incremental change to programs for more effective or efficient delivery. This seems to be bringing evaluation increasingly into the sphere of audit.

Second, restraint in line departments and programs creates an opportunity for review information to become more valuable, if it is applicable to line managers. In addition to identifying the existence of problems, audit is increasingly being asked to investigate the extent and possible solution of problems. On the other hand, evaluation is increasingly asked to comment on the impacts of particular structures or systems on the attainment of objectives. The specific concerns of managers under restraint are leading to the new demands on audit and evaluation groups mentioned previously, and are providing the opportunity for review to win new clients with new (operations-level) products.

For these two reasons, audit has been expanding its scope into performance and program audit, while evaluation is increasingly focusing on process and efficiency questions. As a result, the operational aspects of review are winning out over both the strategic and the accountability features. These developments are not without controversy. Evaluation's leaning toward efficiency, for example, has been a contentious point from the very introduction of the function in Canada, with important voices on each side. The danger is that the contribution that evaluation is uniquely capable of making to the strategic decision and planning processes will be forgone. Similarly, the downgrading of the audit's accountability function in favour of operations is potentially a serious loss. It is not believed that these are the consequences of linking audit and evaluation, but rather aspects of the cause of greater contact between them.

Organizational Linkage

In the late 1980s and early 1990s, most departments moved their audit and evaluation groups into organizational contact with one another. The most common of these structures is to have both reporting to the same Director General of Audit and Evaluation. Other forms include reporting to the same Director or Assistant Deputy Minister, or reporting to an executive who also has responsibility for functions such as management consultancy, planning, or special fraud investigations. The departments studied that had linked their review functions divided fairly evenly between those that had the new unit report to the policy side and to the corporate management side.

In most departments, the motivation to link functions organizationally was to save administrative, overhead, and especially Executive resources. Generally this meant a Director General of Audit and Evaluation responsible for a Director of Audit and a Director of Evaluation, but increasingly in the smaller organizations this is being reduced again to a single Director General with all officers reporting directly to her or him. The result is a flatter, cheaper, and possibly more responsive and efficient organization, but also one that is lower in the corporate hierarchy. This last feature may have a significant impact on the operations of the review organization. A corporate review group with a lower status in the organization will be less in touch with departmental issues, will find it more difficult to find a voice at the strategic level, and may be more allied with program managers. These changes will affect the nature, context, and effectiveness of review activities. On the other hand, it may also mean fewer reporting layers between the review functions and the Deputy.

A different motivation for organizational linkage revolves around what is often referred to as the "critical mass" problem. It is held by some that unless separate audit and evaluation groups are of a certain size, they will not command sufficient respect in the department or have the breadth of skill to provide useful products, and therefore linkage is desirable to enhance the profile and productivity of each function. Several heads of review have cited this as a concern, but it is considered a limited argument, true to the extent that combining very small groups should increase the visibility of both functions within the department. It is not clear why mere organizational linkage of small functions should contribute much to their ability to deliver products.

Because it is usually undertaken for resource reasons, organizational integration is generally not intended by its initiators to have substantive effects on the performance of the audit and evaluation functions. It is expected that business will continue as before in each area, but perhaps more efficiently for having a flatter organization. The possibility is acknowledged of incidental benefits such as sharing information and avoiding overlapping requests to managers, as a result of each side being more aware of the activity of the other. However these are secondary to the desire for reduced overhead and executive costs. Changes in functions from such organizational changes tend to be feared by those involved because it is believed they will weaken the cohesion and skills of each function; organizational linkage tends to be perceived as the thin end of the wedge, representing the beginning of the end for the professional distinction between auditors and evaluators. When the Evaluation and Audit Branch was created in the erstwhile Office of the Comptroller General, substantial effort was necessary to reaffirm to the audit and evaluation communities that the new linkage in the centre did not represent a weakening of the commitment to strong and distinct professional groups, and that the change was an effort to find more effective and efficient means of delivering audit and evaluation products.

It now appears that organizational linkage has in fact had some functional impact on the distinction between audit and evaluation, although not of the sort originally feared. These developments will be discussed in more detail in subsequent sections, but in short they involve the change in perspective of the head attendant upon having one person oversee both functions, the effects of joint planning, the flexibility of resources when one budget covers both functions, and the refinement of methodologies resulting from contact with officers from a related profession.

The "Single Window"

The single window is a particular form of organizational linkage in which clients requiring audit or evaluation services are encouraged to present their questions without regard for the break-down between audit and evaluation issues. This has been found by many departments to be advantageous in that it does not require that clients be informed of the subtleties of the audit and evaluation distinction, and it allows unmeditated and more precise statement by clients of their true needs and interests. For these reasons, the single window is highly valued and desired by most departments.

Half the heads of review contacted explicitly said they sought to provide "single window", "seamless", or "transparent" service to their clients, and only a small number were philosophically committed to having separate audit and evaluation wickets. The latter were the same ones who argued for maintaining organizationally unlinked functions. The applicability of the single window depends on the senior management of the department; some ask for an audit or an evaluation, while others prefer to present their review groups with simply a list of concerns or questions, undifferentiated between audit and evaluation.

Seamless service is independent of organizational integration in that having one does not necessitate having the other, however in practice a single window tends to follow organizational integration. A single point of contact with clients can be a façade in front of separate and well- defined audit and evaluation units, behind which the client's problem is divided into its audit and evaluation components early in the process. The single window, therefore, is only symbolic of a client-orientation on the part of the review group, and may indicate nothing about the internal organization of audit and evaluation.

In addition to serving the goal of client orientation, transparency has a major benefit in that it makes the audit-evaluation distinction a "live" one rather than an artificial one. With clients approaching the audit and evaluation groups with a list of issues or questions to be investigated, it is up to the review functions to assign these to officers with the appropriate skills. This applies equally to audit-evaluation groups that are organizationally separate, linked, or fused, and functionally connected or separate. The process of assigning questions to skills requires that the audit-evaluation line be redrawn each time, based on the realities of the material to hand. Therefore, consciously or not, the organization must satisfy itself each time that its internal structure of audit and evaluation meets the needs of its clients and their studies. This is a valuable exercise in continuous organizational reappraisal.

Functional Linkage

The differences in organization to a large extent mask the fact that the majority of groups do largely the same things. Most heads of audit and/or evaluation agree that reviews fall on a spectrum from audit to evaluation with some grey area in between. This means all departments, regardless of the organizational form of their audit and evaluation functions, perform some traditional audits, some traditional evaluations, and some studies that fall in between. This is true of groups where audit and evaluation are organizationally indistinguishable, as well as those where there is no formal connection between them. At base, most agree that pure audits and pure evaluations are categories of reviews, and that there exists a third category dealing with the grey areas. Functional linkage concerns the management of this grey area.

The term functional linkage encompasses a broad range of activities involving different degrees of audit-evaluation connection, from joint annual planning, to occasional projects involving both audit and evaluation questions, to single-function fused review studies.

The minimum statement of functional linkage is the coordination of annual plans. The degree of "jointness" of joint plans varies. It might be little more than two plan documents being printed together with no substantive interaction having taken place, or vice versa it may be that great cooperation was achieved but the final plans are printed separately. Obviously it is the interaction which is important. A joint annual plan allows the realization of some of the benefits sought by the functionally linked. These include a knowledge of what studies the other group is undertaking, the possibility of sharing information and research, and perhaps some contact with the methods of the other. Joint planning is more or less automatic for organizationally connected groups, but must be consciously sought in departments with independent audit and evaluation units. One head of functionally distinct audit and evaluation groups commented that senior and program manager involvement in the process of a joint annual plan helped sensitize them to the differences between audit and evaluation, and thereby supported the separation of the functions in that department.

It is fairly common, and becoming more so, for a department to engage in occasional joint studies. Approximately half the departments studied have been involved with a joint audit-evaluation venture in the recent past, with another quarter open to the concept although not experienced. The process is generally this: An area containing a potential joint study is generally identified in the annual planning process, following which there is joint exploration of issues of interest, identification of total information needs, and development of question lists and work plans. This is generally put into action by separate investigation and research, possibly with some information exchange if appropriate, and ends with a joint report dealing with the findings of both the audit and evaluation components. This model varies to suit the entity under study, the time needs of the particular investigations, and the preferences of the client and review group. For example, it is not uncommon for the work plans to be developed separately, for some of the execution of the studies to be performed jointly, or for separate reports to be issued. In certain circumstances, a steering committee may be struck composed of representatives of both audit and evaluation. It is important to note that the normal practice in joint ventures is for the audit and evaluation components to be performed quite independently, in a manner similar to a traditional stand-alone audit and evaluation. Planning and reporting tend to be done together, with mainly non-substantive contact along the way. Therefore the "jointness" of most joint studies is not deep.

Few departments even consider moving to the next level of functional linkage where auditors and evaluators are assigned to the same questions or an auditor is sent to investigate what traditionally would be considered an evaluation issue or vice versa. The exception is the fully merged single- function Review group, where this is routine, and this special case will be discussed below. The occasions where departments with distinct audit and evaluation functions used this technique are instructive as they highlight the continuing respect for the fundamental differences between the two. Heads of review reported such "deep cooperation" in the following circumstances: where evaluators helped auditors with client surveys or statistical work; where auditors helped evaluators with cost-effectiveness calculations; where evaluators undertook a major audit because it was felt the "macro perspective" of evaluation was needed; where personnel scheduling or contractor crises arose; and where managers had significantly more respect for one group than the other. In all but the last two, it was the unique strengths of each discipline that made it attractive to assign both auditors and evaluators to the same questions. In the remaining two cases skills were secondary to other considerations, and this was considered sub-optimal but necessary. Therefore, the conclusion can be drawn that the occasions where the professions work together on the same question will tend to reinforce the distinctiveness of each, rather than weaken it. The professions are brought together in such situations, generally to cover the "grey-area" questions involving both audit and evaluation strains, precisely because their different skill sets make it worthwhile to do so.

Several departments choose to do numerous joint ventures but keep audit and evaluation organizationally distinct. This tends to come from the belief that the desire among clients and senior managers for links between audit and evaluation is a cyclical fad, and that the group must protect itself against the pendulum returning to the separateness side by maintaining organizationally distinct but perhaps functionally fused structures.

Single-Function Review

Two or three departments have adopted the philosophy that any formal distinction between audit and evaluation issues or officers is counter productive. These shops are championing Review as a discipline comprising all range of audit and evaluation questions. In theory, this eliminates the cultural impediments to audit and evaluation working together, removes the need for Review clients to understand the differences, and allows a comprehensive view of the operations, management, and results of a program or organization. The approach comes from a conviction that the overlap between audit and evaluation is much greater than is generally accepted, and that therefore it is possible to train one individual to perform both functions adequately. The sceptics have argued it has been used as a means of reforming a problematic function, in that it is hoped associating it with a better-performing group will make the good practices rub off.

Several issues emerge in practice under this regime. First, the question remains as to whether this is merely repackaging the old products, since even the heads of merged Review groups acknowledge that at base their studies could be divided into what would traditionally have been called pure audits, pure evaluations, and "grey-area" studies. The existence of an essential difference between a Review study and the established audit- evaluation products is asserted by some, but has not been demonstrated.

A further concern is that dually-skilled Reviewers may not be as adept at pure audits or pure evaluations as those trained specifically in one or the other. It is often said by those opposed to this model that the head of review should cover the poles of the audit-evaluation spectrum first, then concern him or herself with the grey areas, since if the grey areas are given primacy in the form of Reviewers it may be that studies at the poles will be poorly executed. The occurrence of this in practice is difficult to gauge as these merged groups are relatively new, but those committed to this path are aware of it and do not appear to be dissuaded by it.

 




Impacts of Linkage

With some qualifications, this study has found little to suggest that joint ventures are significantly different from coordinated audit and evaluation studies. It is generally accepted among heads of review representing the whole range of linkage and separation that there remain audit questions, evaluation questions, and "grey area" questions. The fact that auditors and evaluators may be working together on an entity cannot change this, and the questions they ask and results they find appear to be much the same as if they operated independently. This statement must be qualified to take into account the possible effects on stakeholders, on the leadership and reporting of studies, and on the professional development of auditors and evaluators.

Relation to Stakeholders

The review functions have three groups of stakeholders corresponding to the three possible foci of review: program and organizational improvement, strategic management, and accountability to the public. Therefore, the stakeholders to be considered are program managers, senior managers, and Parliament. Some of the interests of these groups will be conflicting because of the various possible objectives of audit and evaluation.

Program managers have generally been in favour of closer coordination, if not outright integration, of the audit and evaluation functions because they are interested in the operational questions that happen to fall where the two overlap. For these questions, maintaining separate functions may seem unproductive. In addition, line managers tend to bear the brunt of the disruption caused by requests by reviewers for information and so have the most to gain from improved coordination of the planning processes.

The attitude of senior management to joint functions seems very much to depend on the personalities and experiences of those leading the department. Some are reportedly psychologically committed to separate audit and evaluation functions, partly due to the previous marketing efforts of the separate groups, while others were never convinced of the utility of the distinction and so are more enthusiastic about the potential of developments such as the single window and joint ventures. It can be suggested that the increased interest in operational and cost-effectiveness information, which is fuelled by the pressures of restraint and is in turn driving the trend toward more connection between audit and evaluation, may impair the ability of the review functions to provide strategic decision- making information to senior management.

Parliament's stake in the audit and evaluation functions concentrates on their accountability aspects, accountability for compliance to codes, for resources and authority dispensed, and for results achieved. The organizational or functional linkage of audit and evaluation does not appear to affect Parliament as a client, as long as the basic accountability capacities of the functions are fulfilled. Other concerns are of greater interest to Parliament, such as the means by which audits and evaluations, however they are organized and labelled, are reported in the Estimates.

The effects on clients are less clear with regard to the impact of moving to a single comprehensive Review discipline. There is the concern that traditional audits and evaluations at the ends of the spectrum may suffer if dually-skilled Reviewers are less talented at these than single-skill auditors and evaluators. Also, the credibility of this function, before program managers, senior managers, and Parliament has not been established, and the investment in building respect and authority for the traditional labels of audit and evaluation on the part of these clients may be lost. This is particularly a concern for audit since it has the longer tradition to uphold.

Relation to the Organization

Organizational linkage of the functions has effects on the groups as organizations. For example, the mission of the new group is redefined in wider terms to encompass both. This may change the mind-set and vision of each group, as the individuals come to see themselves as a part of the wider enterprise of "Corporate Review" rather than just audit or just evaluation. There may be indirect effects on projects and clients from such a development.

Installing a single DG to supervise both the Audit and Evaluation divisions changes the way the head looks at the two functions. It encourages seeing them as a single pool of talent, each individual having particular strengths which are to be matched with the task at hand in an optimal way without restriction by the formal separation of the groups. Therefore, joint studies may become more likely. A second result is that a single window become more likely since access to the functions has been centralized.

Also, the organizational linkage of audit and evaluation may mean they operate from the same budget, increasing the flexibility of allocating resources between them, which is of particular benefit when mid-year crises change the review priorities of the department. However, the Australian National Audit Office has found evidence of the opposite in that small, poorly funded functions may, when linked, aggravate each other's resource problems (ANAO,1990:para.3.3.17).

These effects arising from organizational linkage appear to have been unintended consequences, rather than the conspiracy that many sceptics of linkage anticipated. In any event, it is hard to see these changes in attitude as negative in themselves; they fit well with a client orientation, allowing more a sensitive response to clients needs both at the front end and in terms of available products, and a more flexible use of the skills in the organization. Clients and audit-evaluation managers seem to be happy with this result. It is seen by some however as having dangerous implications for the long run, since it may devalue the professional distinctions between audit and evaluation and encourage clients to disregard the differences between them.

Dangers for the organization from linkage include the possible loss for audit of its traditional independence by being associated with the policy and planning aspects of evaluation. Some departments have continued to benefit from the authoritative connotations of the tern "Audit" by including it in the new group title: for example, the Audit and Review Directorate. Audit may also become more involved in more assessment of controls prior to program implementation, which is both a benefit because it uses the group's skills and a concern since it may compromise the objectivity of a traditional ex-post audit. A concern for the department is that having audit and evaluation report through the same Branch will take one of them out of its traditional reporting path and may reduce the influence of that function. This can only be minimized by careful internal communication.

The hypothesis that organizational linkage is more likely in organizations that have a high proportion of contracted services to in-house officers, while plausible, is not supported by the findings of this study. The theory suggests that, for example, the management of audit contracts can be adequately performed by a comprehensive review organization which does not possess the skills to do the audits itself. There does not appear to be a relation between proportion of contracting and incidence of linkage, probably due to the many factors which combine to create any given level of contracting. However, it does appear to be easier to link audit and evaluation where one or the other relies heavily on contracted services, presumably because there is less cultural resistance to the change.

Joint Venture Leadership and Reporting

Two alternatives present themselves to the problem of the leadership of joint projects. Several departments have experimented with dual leadership by an evaluator and an auditor, but only a few have maintained this practice because of the accountability and territorial problems it tends to create. More successful have been the studies done under the leadership of a single individual, generally from the area most heavily involved in the study, with an advisor or "coach" from the other function.

Functional linkage raises the question of whether it is preferable to report separately or seamlessly. There is no consensus on the question among departments, but in general it is answered on a case-by-case basis determined by the timing of the study components and the needs and sensitivities of the client. Heads of review have found different clients to have different preferences in the area, not necessarily related to the actual content of the report; either they place high value on the traditional labels and so demand separate reports, or they see this as wasteful and prefer a single document. One problem experienced by at least two departments with producing joint reports is the occasion where an evaluation team did background work for an audit, it seemed inadvisable to report jointly since the evaluation findings would have appeared as an appendix to the audit; separate reporting better demonstrated the equal contributions of both functions. In general, it would seem preferable where study components are coming to fruition around the same time, to report jointly so as to maximize the possibility of extra insights emerging by the integration of the two sets of information.

Human Resource Issues

Most review leaders agree that audit and evaluation professionals can learn from each other whether or not they are involved in linked organizations and joint studies. The consensus is that many of the good practices of each discipline can strengthen the other even in its stand-alone activity. The benefits generally cited by heads of review as being most likely or highly prized from cross-disciplinary contact include:

  • widening the perspective of each discipline so that evaluation investigates the operational and management obstacles to desired outcomes and audit places its investigation of the management framework in the context of its impact on the achievement of objectives;
  • changing the form of audit reports, to be more explanatory and easily understood, and opening audit to the possibility of producing several editions of the final report, each tailored to a different audience;
  • inculcating into audit the "data-creating" spirit, the results- orientation, and the critical approach of evaluation;
  • encouraging evaluation to have a sensitivity of the financial performance and consequences of programs, and thereby to become more useful to the budget process.

It does not appear to be enough to expect that individual audit or evaluation officers will be able to master these cross-over skills on their own since even keeping abreast of developments in one discipline is a full- time job. Therefore, some conscious attempt appears necessary to create a situation where this exchange of practices will take place. This can be attempted either through increasing the level of organizational or functional contact between the groups or by engaging in specific training exercises. It is reported by those who have participated in joint ventures that the experience is highly satisfying on this level, and probably more effective than outside training would be. However, the value of this learning is disputed by some heads of review, who estimate that the effort required to gain an incremental unit of improvement in performance from this interdisciplinary learning would be better spent in other forms of training within the individual's specialization.

The separate professional cultures possessed by auditors and evaluators are the subject of much debate. There are those who adamantly believe the existence of distinct cultures is a serious obstacle to maximizing the utility of both functions. They feel the socialization and esprit de corps encourages a narrowness of mind and excludes from consideration many important issues that happen to fall outside the traditional line of business, and that progress cannot be made until the two are thoroughly blended into a "Review" function. On the other hand, there are those who feel the distinct cultures of auditors and evaluators are important as motivating agents for their officers, and for inculcating and defending professional standards, ethics, and methodologies. In particular, audit fears that its long-standing independence from program managers may be compromised by association with the more policy-related evaluation group. Also it is sometimes argued that the groups must be kept separate for the sake of maintaining room for the possibility of career development among auditors and evaluators.

The median opinion seems to be that it is not necessary to radically challenge the cultures in order to gain the benefits from working together, and that on the whole the professional pride and standards inculcated by the cultures are positive for morale and effectiveness. However, it has been found important to encourage smooth the contact between cultures when previously unconnected audit and evaluation groups are brought together. The most common acculturation exercises are branch retreats and introductory courses in the complementary discipline. As one would expect, it has been found less culturally difficult to link organizationally audit and evaluation groups where there was a high ratio of secondees to permanent employees or where there was a high reliance on contract as oppose to in- house work.

Choosing to operate as a single-function "Review" group has significant human resource consequences. Those departments committed to this course have had to invest heavily in acquiring or creating "generalists", officers with the skills and the attitudes required to perform "Reviews". Indeed, the distinct cultures of the two groups, considered by "Review" proponents to be counter-productive, have been found to be sufficiently ingrained that it is easier to hire program managers or outsiders with no audit-evaluation training and teach them the required skills than to break down the socialization of either auditors or evaluators. This may suggest that single-function Review groups will perform below the level of their non- fused counterparts, at least in a transition period until those officers brought in from outside become experienced.

 




Conclusions

Organizational linkage appears to have some significant probable benefits, with relatively modest dangers. In addition to the potential resource savings which have been the motivation for senior management, the benefits include facilitating a client orientation in the form of a single window and the opportunity for each side to learn from the methods of the other. The danger is that individuals socialized in the separate professions will feel threatened by the proximity of a different culture and assume it reflects senior management's ignorance of the differences between the functions. The fact that organizational linkage has tended to follow cuts to the functions contributes to this negative association.

Approximately half the departments visited have performed a joint study between audit and evaluation in the recent past, with another quarter open to the possibility of doing so in the near future. In most of these cases, however, the process of a joint venture resembles separate but coordinated audit and evaluation studies. The traditional division of questions between the functions is generally respected.

Fusion into a single-function Review group carries potentially more serious negative consequences, and tends to be initiated for quite different reasons that the two preceding forms of linkage. It is too early to assess whether generalist "Reviewers" produce studies which are qualitatively different from those of auditors and evaluators, but the front-end costs in terms of training and acculturation appear quite high.

 




Individuals Interviewed for the Study

(Positions and department names at time of contact)

  1. Jim van Adel, Acting Director General, Internal Audit, National Defence.
  2. Robert Bergeron, Director, Corporate Review and Audit, Fisheries and Oceans.
  3. Barry Bragg, Acting Director General, Audit and Evaluation Branch, Public Works Canada.
  4. Francine Chabot-Plante, Director General, Audit and Evaluation, Communications Canada.
  5. Bill Clark, Acting Director General, Departmental Audit Branch, Indian and Northern Affaris Canada.
  6. David Bell, Assistant Deputy Minister - Review, Transport Canada.
  7. Elizabeth Hopkins, Director, Evaluation, Supply and Services Canada.
  8. Nicole Mendenhall, Director General, Audit and Evaluation, Revenue Canada - Customs and Excise.
  9. Basil Orsini, Director, Audit and Evaluation, Revenue Canada - Taxation.
  10. Denise Roberge, Director, Internal Audit and Program Evaluation, Labour Canada.
  11. Bill Smith, Director, Corporate Planning and Evaluation, National Research Council.
  12. Owen Taylor, Director General, Operations Audit Branch, Industry and Science Canada.
  13. Steven Turner, Director General, Review Branch, Supply and Services Canada.
  14. Al Whitla, Director, Audit and Evaluation, External Affairs and International Trade Canada.
  15. Steve Wilson, Auditor, Correctional Services Canada.

 




References

The following are selected references on linkages between internal audit and evaluation.

  1. Chelimsky, E (1985), "Comparing and Contrasting Auditing and Evaluation: some Notes on their Relationship", Evaluation Review, 9 (4), 483-503.
  2. Comptroller General of the United States (1988), Government Auditing Standards, United States General Accounting Office.
  3. Davis, D. F. (1990), "Do you want a Performance Audit or a Program Evaluation?", Public Administration Review, 50 (1), 35-41.
  4. Hendricks M., Mangano M.F. & Moran W.C. (1990), "Inspector Generals; A New Force in Evaluation", New Directions for Program Evaluation, 48.
  5. Management Advisory Board-Management Improvement Advisory Committee, Australia (1993), Performance Information and the Management Cycle, Publication no 10.
  6. Office of the Comptroller General (1991), Line Managers and Asssessing Service to the Public.
  7. Office of the Comptroller General (1992), Assessing Performance: A Managers Guide to Program Evaluation and Internal Audit.
  8. Rist, R.C. (1989), "Management Accountability: the Signals Sent by Auditing and Evaluation", Journal of Public Policy, 9 (3), 355-369.
  9. Segsworth R. V. (1990), Canadian Journal of Program Evaluation, 5 (1),41- 56, "Auditing and Evaluation in the Government of Canada: Some Reflections".
  10. Walker W.E. (1988), Evaluation and Program Planning, 8, 271-280, "The conduct of program evaluation reviews in the GAO".
  11. Walker W.E. (1988), Evaluation and Program Planning, 8, 359-366, "The impact of program evaluations on government".