ARCHIVED - Linkages between Audit and Evaluation in Canadian Federal Departments (TBS Paper) - September 30, 1993

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

Forms of Linkage

History, Trends, and Theory

Audit and evaluation developed quite separately, at different times, and out of different disciplines. Audit has its history in accounting and financial management, and adopted their traditional independence from program management. It gets much of its strength from the fact that it has a largely agreed upon set of standards (IIA and federal). It delivers a range of products, from comprehensive to compliance audits, dealing with different aspects of an organization, but in all forms it takes as its starting point the management and operations of the organization, and moves outward to the organization's activities and products.

Evaluation, with its more recent heritage in the 1960s, arose in response to a perceived need in the United States for an understanding of the actual effects of military "Great Society" projects. It uses (largely social-) scientific methods, providing empirical evidence of the impacts and effects of public policies, intended to be used as inputs to decision making. Evaluation begins at the impacts and outputs of the organization and may move inward to examine the operational reasons for these results. Evaluation has developed its own set of standards (AEA and federal) for measuring performance, but it maintains a traditional connection with program managers and planners, valuing objectivity over independence in practice.

Reflecting the origins of each function, audit in the federal government has traditionally been attached to the financial management area and evaluation to policy development. While it is generally acknowledged that audit and evaluation form a continuum of organizational information services with overlap in areas such as efficiency of operations and cost effectiveness, they were traditionally kept separate, with evaluation concerned with analysis of policy and outputs, and auditing with internal financial controls and management systems.

These traditions have been changing in the last decade in many jurisdictions and in both private and public organizations. There appear to be two reasons for this in the public sector, both related to restraint. First, restraint in the audit and evaluation sectors means a new pressure on the functions to show their worth by meeting client demands for products. This requires flexibility in defining the list of available products, and a willingness to take on tasks which do not fit the traditional disciplinary boundaries. It appears the demand among clients is changing in the direction of the products providing information on operations, with the objective of incremental change to programs for more effective or efficient delivery. This seems to be bringing evaluation increasingly into the sphere of audit.

Second, restraint in line departments and programs creates an opportunity for review information to become more valuable, if it is applicable to line managers. In addition to identifying the existence of problems, audit is increasingly being asked to investigate the extent and possible solution of problems. On the other hand, evaluation is increasingly asked to comment on the impacts of particular structures or systems on the attainment of objectives. The specific concerns of managers under restraint are leading to the new demands on audit and evaluation groups mentioned previously, and are providing the opportunity for review to win new clients with new (operations-level) products.

For these two reasons, audit has been expanding its scope into performance and program audit, while evaluation is increasingly focusing on process and efficiency questions. As a result, the operational aspects of review are winning out over both the strategic and the accountability features. These developments are not without controversy. Evaluation's leaning toward efficiency, for example, has been a contentious point from the very introduction of the function in Canada, with important voices on each side. The danger is that the contribution that evaluation is uniquely capable of making to the strategic decision and planning processes will be forgone. Similarly, the downgrading of the audit's accountability function in favour of operations is potentially a serious loss. It is not believed that these are the consequences of linking audit and evaluation, but rather aspects of the cause of greater contact between them.

Organizational Linkage

In the late 1980s and early 1990s, most departments moved their audit and evaluation groups into organizational contact with one another. The most common of these structures is to have both reporting to the same Director General of Audit and Evaluation. Other forms include reporting to the same Director or Assistant Deputy Minister, or reporting to an executive who also has responsibility for functions such as management consultancy, planning, or special fraud investigations. The departments studied that had linked their review functions divided fairly evenly between those that had the new unit report to the policy side and to the corporate management side.

In most departments, the motivation to link functions organizationally was to save administrative, overhead, and especially Executive resources. Generally this meant a Director General of Audit and Evaluation responsible for a Director of Audit and a Director of Evaluation, but increasingly in the smaller organizations this is being reduced again to a single Director General with all officers reporting directly to her or him. The result is a flatter, cheaper, and possibly more responsive and efficient organization, but also one that is lower in the corporate hierarchy. This last feature may have a significant impact on the operations of the review organization. A corporate review group with a lower status in the organization will be less in touch with departmental issues, will find it more difficult to find a voice at the strategic level, and may be more allied with program managers. These changes will affect the nature, context, and effectiveness of review activities. On the other hand, it may also mean fewer reporting layers between the review functions and the Deputy.

A different motivation for organizational linkage revolves around what is often referred to as the "critical mass" problem. It is held by some that unless separate audit and evaluation groups are of a certain size, they will not command sufficient respect in the department or have the breadth of skill to provide useful products, and therefore linkage is desirable to enhance the profile and productivity of each function. Several heads of review have cited this as a concern, but it is considered a limited argument, true to the extent that combining very small groups should increase the visibility of both functions within the department. It is not clear why mere organizational linkage of small functions should contribute much to their ability to deliver products.

Because it is usually undertaken for resource reasons, organizational integration is generally not intended by its initiators to have substantive effects on the performance of the audit and evaluation functions. It is expected that business will continue as before in each area, but perhaps more efficiently for having a flatter organization. The possibility is acknowledged of incidental benefits such as sharing information and avoiding overlapping requests to managers, as a result of each side being more aware of the activity of the other. However these are secondary to the desire for reduced overhead and executive costs. Changes in functions from such organizational changes tend to be feared by those involved because it is believed they will weaken the cohesion and skills of each function; organizational linkage tends to be perceived as the thin end of the wedge, representing the beginning of the end for the professional distinction between auditors and evaluators. When the Evaluation and Audit Branch was created in the erstwhile Office of the Comptroller General, substantial effort was necessary to reaffirm to the audit and evaluation communities that the new linkage in the centre did not represent a weakening of the commitment to strong and distinct professional groups, and that the change was an effort to find more effective and efficient means of delivering audit and evaluation products.

It now appears that organizational linkage has in fact had some functional impact on the distinction between audit and evaluation, although not of the sort originally feared. These developments will be discussed in more detail in subsequent sections, but in short they involve the change in perspective of the head attendant upon having one person oversee both functions, the effects of joint planning, the flexibility of resources when one budget covers both functions, and the refinement of methodologies resulting from contact with officers from a related profession.

The "Single Window"

The single window is a particular form of organizational linkage in which clients requiring audit or evaluation services are encouraged to present their questions without regard for the break-down between audit and evaluation issues. This has been found by many departments to be advantageous in that it does not require that clients be informed of the subtleties of the audit and evaluation distinction, and it allows unmeditated and more precise statement by clients of their true needs and interests. For these reasons, the single window is highly valued and desired by most departments.

Half the heads of review contacted explicitly said they sought to provide "single window", "seamless", or "transparent" service to their clients, and only a small number were philosophically committed to having separate audit and evaluation wickets. The latter were the same ones who argued for maintaining organizationally unlinked functions. The applicability of the single window depends on the senior management of the department; some ask for an audit or an evaluation, while others prefer to present their review groups with simply a list of concerns or questions, undifferentiated between audit and evaluation.

Seamless service is independent of organizational integration in that having one does not necessitate having the other, however in practice a single window tends to follow organizational integration. A single point of contact with clients can be a façade in front of separate and well- defined audit and evaluation units, behind which the client's problem is divided into its audit and evaluation components early in the process. The single window, therefore, is only symbolic of a client-orientation on the part of the review group, and may indicate nothing about the internal organization of audit and evaluation.

In addition to serving the goal of client orientation, transparency has a major benefit in that it makes the audit-evaluation distinction a "live" one rather than an artificial one. With clients approaching the audit and evaluation groups with a list of issues or questions to be investigated, it is up to the review functions to assign these to officers with the appropriate skills. This applies equally to audit-evaluation groups that are organizationally separate, linked, or fused, and functionally connected or separate. The process of assigning questions to skills requires that the audit-evaluation line be redrawn each time, based on the realities of the material to hand. Therefore, consciously or not, the organization must satisfy itself each time that its internal structure of audit and evaluation meets the needs of its clients and their studies. This is a valuable exercise in continuous organizational reappraisal.

Functional Linkage

The differences in organization to a large extent mask the fact that the majority of groups do largely the same things. Most heads of audit and/or evaluation agree that reviews fall on a spectrum from audit to evaluation with some grey area in between. This means all departments, regardless of the organizational form of their audit and evaluation functions, perform some traditional audits, some traditional evaluations, and some studies that fall in between. This is true of groups where audit and evaluation are organizationally indistinguishable, as well as those where there is no formal connection between them. At base, most agree that pure audits and pure evaluations are categories of reviews, and that there exists a third category dealing with the grey areas. Functional linkage concerns the management of this grey area.

The term functional linkage encompasses a broad range of activities involving different degrees of audit-evaluation connection, from joint annual planning, to occasional projects involving both audit and evaluation questions, to single-function fused review studies.

The minimum statement of functional linkage is the coordination of annual plans. The degree of "jointness" of joint plans varies. It might be little more than two plan documents being printed together with no substantive interaction having taken place, or vice versa it may be that great cooperation was achieved but the final plans are printed separately. Obviously it is the interaction which is important. A joint annual plan allows the realization of some of the benefits sought by the functionally linked. These include a knowledge of what studies the other group is undertaking, the possibility of sharing information and research, and perhaps some contact with the methods of the other. Joint planning is more or less automatic for organizationally connected groups, but must be consciously sought in departments with independent audit and evaluation units. One head of functionally distinct audit and evaluation groups commented that senior and program manager involvement in the process of a joint annual plan helped sensitize them to the differences between audit and evaluation, and thereby supported the separation of the functions in that department.

It is fairly common, and becoming more so, for a department to engage in occasional joint studies. Approximately half the departments studied have been involved with a joint audit-evaluation venture in the recent past, with another quarter open to the concept although not experienced. The process is generally this: An area containing a potential joint study is generally identified in the annual planning process, following which there is joint exploration of issues of interest, identification of total information needs, and development of question lists and work plans. This is generally put into action by separate investigation and research, possibly with some information exchange if appropriate, and ends with a joint report dealing with the findings of both the audit and evaluation components. This model varies to suit the entity under study, the time needs of the particular investigations, and the preferences of the client and review group. For example, it is not uncommon for the work plans to be developed separately, for some of the execution of the studies to be performed jointly, or for separate reports to be issued. In certain circumstances, a steering committee may be struck composed of representatives of both audit and evaluation. It is important to note that the normal practice in joint ventures is for the audit and evaluation components to be performed quite independently, in a manner similar to a traditional stand-alone audit and evaluation. Planning and reporting tend to be done together, with mainly non-substantive contact along the way. Therefore the "jointness" of most joint studies is not deep.

Few departments even consider moving to the next level of functional linkage where auditors and evaluators are assigned to the same questions or an auditor is sent to investigate what traditionally would be considered an evaluation issue or vice versa. The exception is the fully merged single- function Review group, where this is routine, and this special case will be discussed below. The occasions where departments with distinct audit and evaluation functions used this technique are instructive as they highlight the continuing respect for the fundamental differences between the two. Heads of review reported such "deep cooperation" in the following circumstances: where evaluators helped auditors with client surveys or statistical work; where auditors helped evaluators with cost-effectiveness calculations; where evaluators undertook a major audit because it was felt the "macro perspective" of evaluation was needed; where personnel scheduling or contractor crises arose; and where managers had significantly more respect for one group than the other. In all but the last two, it was the unique strengths of each discipline that made it attractive to assign both auditors and evaluators to the same questions. In the remaining two cases skills were secondary to other considerations, and this was considered sub-optimal but necessary. Therefore, the conclusion can be drawn that the occasions where the professions work together on the same question will tend to reinforce the distinctiveness of each, rather than weaken it. The professions are brought together in such situations, generally to cover the "grey-area" questions involving both audit and evaluation strains, precisely because their different skill sets make it worthwhile to do so.

Several departments choose to do numerous joint ventures but keep audit and evaluation organizationally distinct. This tends to come from the belief that the desire among clients and senior managers for links between audit and evaluation is a cyclical fad, and that the group must protect itself against the pendulum returning to the separateness side by maintaining organizationally distinct but perhaps functionally fused structures.

Single-Function Review

Two or three departments have adopted the philosophy that any formal distinction between audit and evaluation issues or officers is counter productive. These shops are championing Review as a discipline comprising all range of audit and evaluation questions. In theory, this eliminates the cultural impediments to audit and evaluation working together, removes the need for Review clients to understand the differences, and allows a comprehensive view of the operations, management, and results of a program or organization. The approach comes from a conviction that the overlap between audit and evaluation is much greater than is generally accepted, and that therefore it is possible to train one individual to perform both functions adequately. The sceptics have argued it has been used as a means of reforming a problematic function, in that it is hoped associating it with a better-performing group will make the good practices rub off.

Several issues emerge in practice under this regime. First, the question remains as to whether this is merely repackaging the old products, since even the heads of merged Review groups acknowledge that at base their studies could be divided into what would traditionally have been called pure audits, pure evaluations, and "grey-area" studies. The existence of an essential difference between a Review study and the established audit- evaluation products is asserted by some, but has not been demonstrated.

A further concern is that dually-skilled Reviewers may not be as adept at pure audits or pure evaluations as those trained specifically in one or the other. It is often said by those opposed to this model that the head of review should cover the poles of the audit-evaluation spectrum first, then concern him or herself with the grey areas, since if the grey areas are given primacy in the form of Reviewers it may be that studies at the poles will be poorly executed. The occurrence of this in practice is difficult to gauge as these merged groups are relatively new, but those committed to this path are aware of it and do not appear to be dissuaded by it.

 

• Previous Page
• Table of Contents
• Next Page