ARCHIVED - Linkages between Audit and Evaluation in Canadian Federal Departments (TBS Paper) - September 30, 1993

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

Impacts of Linkage

With some qualifications, this study has found little to suggest that joint ventures are significantly different from coordinated audit and evaluation studies. It is generally accepted among heads of review representing the whole range of linkage and separation that there remain audit questions, evaluation questions, and "grey area" questions. The fact that auditors and evaluators may be working together on an entity cannot change this, and the questions they ask and results they find appear to be much the same as if they operated independently. This statement must be qualified to take into account the possible effects on stakeholders, on the leadership and reporting of studies, and on the professional development of auditors and evaluators.

Relation to Stakeholders

The review functions have three groups of stakeholders corresponding to the three possible foci of review: program and organizational improvement, strategic management, and accountability to the public. Therefore, the stakeholders to be considered are program managers, senior managers, and Parliament. Some of the interests of these groups will be conflicting because of the various possible objectives of audit and evaluation.

Program managers have generally been in favour of closer coordination, if not outright integration, of the audit and evaluation functions because they are interested in the operational questions that happen to fall where the two overlap. For these questions, maintaining separate functions may seem unproductive. In addition, line managers tend to bear the brunt of the disruption caused by requests by reviewers for information and so have the most to gain from improved coordination of the planning processes.

The attitude of senior management to joint functions seems very much to depend on the personalities and experiences of those leading the department. Some are reportedly psychologically committed to separate audit and evaluation functions, partly due to the previous marketing efforts of the separate groups, while others were never convinced of the utility of the distinction and so are more enthusiastic about the potential of developments such as the single window and joint ventures. It can be suggested that the increased interest in operational and cost-effectiveness information, which is fuelled by the pressures of restraint and is in turn driving the trend toward more connection between audit and evaluation, may impair the ability of the review functions to provide strategic decision- making information to senior management.

Parliament's stake in the audit and evaluation functions concentrates on their accountability aspects, accountability for compliance to codes, for resources and authority dispensed, and for results achieved. The organizational or functional linkage of audit and evaluation does not appear to affect Parliament as a client, as long as the basic accountability capacities of the functions are fulfilled. Other concerns are of greater interest to Parliament, such as the means by which audits and evaluations, however they are organized and labelled, are reported in the Estimates.

The effects on clients are less clear with regard to the impact of moving to a single comprehensive Review discipline. There is the concern that traditional audits and evaluations at the ends of the spectrum may suffer if dually-skilled Reviewers are less talented at these than single-skill auditors and evaluators. Also, the credibility of this function, before program managers, senior managers, and Parliament has not been established, and the investment in building respect and authority for the traditional labels of audit and evaluation on the part of these clients may be lost. This is particularly a concern for audit since it has the longer tradition to uphold.

Relation to the Organization

Organizational linkage of the functions has effects on the groups as organizations. For example, the mission of the new group is redefined in wider terms to encompass both. This may change the mind-set and vision of each group, as the individuals come to see themselves as a part of the wider enterprise of "Corporate Review" rather than just audit or just evaluation. There may be indirect effects on projects and clients from such a development.

Installing a single DG to supervise both the Audit and Evaluation divisions changes the way the head looks at the two functions. It encourages seeing them as a single pool of talent, each individual having particular strengths which are to be matched with the task at hand in an optimal way without restriction by the formal separation of the groups. Therefore, joint studies may become more likely. A second result is that a single window become more likely since access to the functions has been centralized.

Also, the organizational linkage of audit and evaluation may mean they operate from the same budget, increasing the flexibility of allocating resources between them, which is of particular benefit when mid-year crises change the review priorities of the department. However, the Australian National Audit Office has found evidence of the opposite in that small, poorly funded functions may, when linked, aggravate each other's resource problems (ANAO,1990:para.3.3.17).

These effects arising from organizational linkage appear to have been unintended consequences, rather than the conspiracy that many sceptics of linkage anticipated. In any event, it is hard to see these changes in attitude as negative in themselves; they fit well with a client orientation, allowing more a sensitive response to clients needs both at the front end and in terms of available products, and a more flexible use of the skills in the organization. Clients and audit-evaluation managers seem to be happy with this result. It is seen by some however as having dangerous implications for the long run, since it may devalue the professional distinctions between audit and evaluation and encourage clients to disregard the differences between them.

Dangers for the organization from linkage include the possible loss for audit of its traditional independence by being associated with the policy and planning aspects of evaluation. Some departments have continued to benefit from the authoritative connotations of the tern "Audit" by including it in the new group title: for example, the Audit and Review Directorate. Audit may also become more involved in more assessment of controls prior to program implementation, which is both a benefit because it uses the group's skills and a concern since it may compromise the objectivity of a traditional ex-post audit. A concern for the department is that having audit and evaluation report through the same Branch will take one of them out of its traditional reporting path and may reduce the influence of that function. This can only be minimized by careful internal communication.

The hypothesis that organizational linkage is more likely in organizations that have a high proportion of contracted services to in-house officers, while plausible, is not supported by the findings of this study. The theory suggests that, for example, the management of audit contracts can be adequately performed by a comprehensive review organization which does not possess the skills to do the audits itself. There does not appear to be a relation between proportion of contracting and incidence of linkage, probably due to the many factors which combine to create any given level of contracting. However, it does appear to be easier to link audit and evaluation where one or the other relies heavily on contracted services, presumably because there is less cultural resistance to the change.

Joint Venture Leadership and Reporting

Two alternatives present themselves to the problem of the leadership of joint projects. Several departments have experimented with dual leadership by an evaluator and an auditor, but only a few have maintained this practice because of the accountability and territorial problems it tends to create. More successful have been the studies done under the leadership of a single individual, generally from the area most heavily involved in the study, with an advisor or "coach" from the other function.

Functional linkage raises the question of whether it is preferable to report separately or seamlessly. There is no consensus on the question among departments, but in general it is answered on a case-by-case basis determined by the timing of the study components and the needs and sensitivities of the client. Heads of review have found different clients to have different preferences in the area, not necessarily related to the actual content of the report; either they place high value on the traditional labels and so demand separate reports, or they see this as wasteful and prefer a single document. One problem experienced by at least two departments with producing joint reports is the occasion where an evaluation team did background work for an audit, it seemed inadvisable to report jointly since the evaluation findings would have appeared as an appendix to the audit; separate reporting better demonstrated the equal contributions of both functions. In general, it would seem preferable where study components are coming to fruition around the same time, to report jointly so as to maximize the possibility of extra insights emerging by the integration of the two sets of information.

Human Resource Issues

Most review leaders agree that audit and evaluation professionals can learn from each other whether or not they are involved in linked organizations and joint studies. The consensus is that many of the good practices of each discipline can strengthen the other even in its stand-alone activity. The benefits generally cited by heads of review as being most likely or highly prized from cross-disciplinary contact include:

• widening the perspective of each discipline so that evaluation investigates the operational and management obstacles to desired outcomes and audit places its investigation of the management framework in the context of its impact on the achievement of objectives;
• changing the form of audit reports, to be more explanatory and easily understood, and opening audit to the possibility of producing several editions of the final report, each tailored to a different audience;
• inculcating into audit the "data-creating" spirit, the results- orientation, and the critical approach of evaluation;
• encouraging evaluation to have a sensitivity of the financial performance and consequences of programs, and thereby to become more useful to the budget process.

It does not appear to be enough to expect that individual audit or evaluation officers will be able to master these cross-over skills on their own since even keeping abreast of developments in one discipline is a full- time job. Therefore, some conscious attempt appears necessary to create a situation where this exchange of practices will take place. This can be attempted either through increasing the level of organizational or functional contact between the groups or by engaging in specific training exercises. It is reported by those who have participated in joint ventures that the experience is highly satisfying on this level, and probably more effective than outside training would be. However, the value of this learning is disputed by some heads of review, who estimate that the effort required to gain an incremental unit of improvement in performance from this interdisciplinary learning would be better spent in other forms of training within the individual's specialization.

The separate professional cultures possessed by auditors and evaluators are the subject of much debate. There are those who adamantly believe the existence of distinct cultures is a serious obstacle to maximizing the utility of both functions. They feel the socialization and esprit de corps encourages a narrowness of mind and excludes from consideration many important issues that happen to fall outside the traditional line of business, and that progress cannot be made until the two are thoroughly blended into a "Review" function. On the other hand, there are those who feel the distinct cultures of auditors and evaluators are important as motivating agents for their officers, and for inculcating and defending professional standards, ethics, and methodologies. In particular, audit fears that its long-standing independence from program managers may be compromised by association with the more policy-related evaluation group. Also it is sometimes argued that the groups must be kept separate for the sake of maintaining room for the possibility of career development among auditors and evaluators.

The median opinion seems to be that it is not necessary to radically challenge the cultures in order to gain the benefits from working together, and that on the whole the professional pride and standards inculcated by the cultures are positive for morale and effectiveness. However, it has been found important to encourage smooth the contact between cultures when previously unconnected audit and evaluation groups are brought together. The most common acculturation exercises are branch retreats and introductory courses in the complementary discipline. As one would expect, it has been found less culturally difficult to link organizationally audit and evaluation groups where there was a high ratio of secondees to permanent employees or where there was a high reliance on contract as oppose to in- house work.

Choosing to operate as a single-function "Review" group has significant human resource consequences. Those departments committed to this course have had to invest heavily in acquiring or creating "generalists", officers with the skills and the attitudes required to perform "Reviews". Indeed, the distinct cultures of the two groups, considered by "Review" proponents to be counter-productive, have been found to be sufficiently ingrained that it is easier to hire program managers or outsiders with no audit-evaluation training and teach them the required skills than to break down the socialization of either auditors or evaluators. This may suggest that single-function Review groups will perform below the level of their non- fused counterparts, at least in a transition period until those officers brought in from outside become experienced.

 

• Previous Page
• Table of Contents
• Next Page