Five-Year Evaluation of the 2006 Policy on Internal Audit

Acknowledgements

This evaluation was undertaken by Ference Weicker & Co. on behalf of the Office of the Comptroller General. This report has been edited for publication in accordance with the publication and communications quality assurance processes within the Treasury Board of Canada Secretariat.

Table of Contents

Executive Summary

Purpose of Evaluation

This evaluation was undertaken as per the requirements of section 5.8.6 of the Policy on Internal Audit and as part of the Secretariat’s Five-Year Evaluation Plan. The objective of the evaluation was to assess the relevance and performance of the Policy on Internal Audit. The evaluation covers the period from April 1, 2006, to March 31, 2010. The scope of the evaluation includes the 2006 Policy on Internal Audit and adjustments made to this Policy as of July 1, 2009.

Findings and Conclusions

Overall, the 2006 Policy on Internal Audit has achieved significant impacts and there exists widespread satisfaction with the Policy. The key findings and conclusions of this evaluation are outlined as follows.

  1. There is a continued need for the Policy.

Although there have not recently been any high-profile breakdowns of control in the federal government, there is still a need for the Policy. The importance of a risk-based internal audit function has increased as a result of the Federal Accountability Act and the new role of deputy heads as the accounting officers of their respective organizations.

In addition, by addressing a number of weaknesses, the Policy has resulted in the following:

  • Independence of departmental audit committees (DACs);
  • Independence of internal audit staff from line management;
  • Sufficient focus on assurance services; and
  • Consistency in internal audit capacity, skills and practices.
  1. The Policy has contributed to increasing the independence of the internal audit function from line management. The Policy requires DACs to have a majority of external members, as well as a chief audit executive (CAE) who reports directly to the deputy head of a large department or agency (LDA).

Prior to the Policy, DACs consisted primarily of senior managers from the relevant department. As a result of the Policy, DACs now have external members who are selected for their extensive experience in the public and private sectors and whose goal is to provide objective, independent observations to DACs. In addition, the direct reporting of CAEs to deputy heads and the direct access that CAEs have to the DACs have increased the independence of the internal audit function from line management.

  1. The implementation of the Policy, particularly the establishment of DACs that have external members, has contributed to improving risk management, governance, internal control and stewardship of resources in federal government departments and agencies.

The key factors that have helped achieve these intended outcomes include the following:

  • Strong deputy head support for Policy implementation;
  • The establishment of a more professional internal audit function within departments;
  • Improved oversight of the internal audit function by a qualified CAE and DAC members who have audit expertise; and
  • More timely implementation of internal audit recommendations, leading to improved management and business practices.

The Policy’s contribution is important because the Policy is complementary to a number of other federal government policies and initiatives that focus directly on areas such as internal control and risk management.

  1. Deputy heads of LDAs have confidence in the assurance provided by the internal audit function and the advice provided by their DAC on risk management, control and governance processes.

Given the depth and breadth of experience held by DAC members, many deputy heads stated that they use the DAC as a sounding board and a strategic resource for strengthening the overall institution, rather than just the internal audit function.

  1. Implementation of the Policy has significantly increased the effectiveness of the internal audit function across government.

The establishment of DACs that have external members has contributed to the effectiveness of the internal audit function within departments. Other contributing factors include the following:

  • The establishment of a CAE position reporting directly to the deputy head;
  • Strong deputy head support for Policy implementation;
  • Incremental funding for departmental internal audit activities;
  • Implementation of Government of Canada audit standards; and
  • Horizontal audits of small department and agencies (SDAs) conducted by the Office of the Comptroller General (OCG).

The proportion of LDAs that obtained an acceptable or strong Management Accountability Framework (MAF) rating for their internal audit function more than doubled, from 42 per cent in 2005–06 to 85 per cent in 2009–10.

  1. Implementation of the Policy has resulted in a considerable increase in management action on internal audit report recommendations due primarily to a concerted focus by DACs and CAEs on following up on the implementation of management action plans.

Since the Policy was introduced, there has been more formal tracking of the implementation of internal audit recommendations by CAEs and internal audit staff. Although difficult to quantify, some benefits that have resulted from the implementation of management action plans include averted risks, improved operational efficiency, and improved risk management and cost savings.

  1. The Policy has contributed to increasing the capacity of the internal audit function across government through the approval of $40 million annually in accompanying funding for departmental and OCG internal audit activities.

With the removal of two budgeted expenditures that have not yet been incurred ($9.7 million per year for increased departmental internal audit salaries projected to occur from reconfiguration of the internal audit community, and $1.3 million per year for technological support), the amount approved on an ongoing basis for other departmental and OCG audit activities is approximately $29.4 million per year. This funding has contributed to an increase of 75 per cent in departmental internal audit expenditures, from $47.2 million in 2005–06 to $82.6 million in 2009–10, in the 41 LDAs that provided financial information. The number of staff devoted to the internal audit function, including the administrative staff who support the DACs and CAEs, has increased by 68 per cent, from 359 FTEsSee footnote 1 in 2005–06 to 602 FTEs in 2009–10, in the LDAs surveyed.

  1. The Policy has contributed to audit coverage appropriate to the level of risk across government through the provision of accompanying funding to bolster departmental internal audit services, horizontal audits of LDAs and SDAs conducted by the OCG, and the preparation of risk-based audit plans that focus on high risk areas.

There are drawbacks to the current criteria for determining whether an organization should be classified as an LDA or an SDA (i.e., 500 FTEs and $300 million in annual expenditures). Some of these drawbacks include the following:

  • The criteria do not reflect the characteristics of micro-agencies, which are very small (they account for less than 0.1 per cent of federal government expenditures) and have difficulty meeting the current reporting and other requirements for SDAs.
  • The criteria do not take into account the level of risk within an organization in determining whether the organization should be classified as an LDA or SDA.
  • The criteria do not align with the criteria used in other Treasury Board policies to separate federal government departments and agencies into large and small organizations.
  1. The Policy has increased the professionalism of the internal audit function by adding rigour to the internal audit standards and practices used by the federal government and by ensuring a more certified internal audit function across government.

The Policy has led to a stronger emphasis on meeting Government of Canada internal audit standards. Training expenditures have increased threefold, from $0.7 million in 2005–06 to $2.1 million in 2009–10, in the 23 LDAs that provided information on their training expenditures. The number of CAEs who have a Certified Internal Auditor (CIA) designation has increased from 12 in 2008 to 19 in 2010; another 11 CAEs are in the process of obtaining a CIA, which, when completed, will mean that approximately 62 per cent of CAEs have a CIA designation. The number of internal audit staff in the federal government who have a CIA designation has almost doubled, from 57 in 2008 to 105 in 2010.

  1. The Policy has been implemented as intended, but some aspects have not been implemented fully.

Some aspects of the Policy that have not been fully implemented are:

  • The reconfiguration of the internal audit community (including the reclassification of internal audit staff);
  • Practice inspections and external assessments; and
  • Annual overview reporting and technological support for standardizing internal audit work processes.

As previously indicated, the $35.4 million increase in internal audit expenditures from 2005–06 to 2009–10 in the 41 LDAs surveyed is greater than the $24 million provided annually under the Policy to these LDAs for their incremental internal audit expenses. Approximately 71 per cent of the LDAs surveyed spent more on their internal audit function than the incremental funding provided to them; the remaining 29 per cent spent less than the incremental funding they received in 2009–10.

The 2009 changes to the Policy were appropriate and have facilitated implementation of the Policy. The changes clarified a number of areas that have been the subject of discussion since the Policy came into force in 2006 (e.g., DAC advisory role, removal of holistic opinion, Minister’s involvement with DAC members) without changing the principles of the Policy. Additional clarification is needed on how to implement the recent guidelines on annual overview reporting.

  1. The activities carried out by the OCG to help organizations implement the Policy have been appropriate for the effective implementation of the Policy, but some have not been very timely; however, recent improvements have been made.

The three OCG activities rated as most useful by CAEs and internal audit staff are DAC training and networking, guidance on internal audit standards and practices, and guidance provided to CAEs. The three activities rated as least useful by both CAEs and internal audit staff are HR strategies and staffing, omnibus supply arrangements (PASS), and horizontal audits of LDAs.

  1. The MAF rating system is the key method employed by the OCG to monitor the effectiveness and impacts of the activities related to the Policy. Respondents indicated that this system is a somewhat useful means of assessing the performance of the internal audit function within LDAs.

On average, LDA deputy heads, CAEs and internal audit staff stated that MAF ratings are between “somewhat useful” and “useful” as a measure of the performance of their departmental internal audit function. Some respondents indicated that MAF criteria have improved recently. Several respondents indicated that MAF requirements impose a heavy reporting burden, particularly on small LDAs.

  1. Policy implementation resource levels are appropriate due to the approval of ongoing accompanying funding to implement the Policy.

This accompanying funding has enabled federal government departments and agencies to restore the capacity of their internal audit function, which was reduced in the 1990s. Based on a comparison with other jurisdictions, the ratio between the number of internal auditors and government expenditures in the Government of Canada is similar to the ratios in the governments of the United Kingdom and Ontario.

  1. The Policy has been implemented efficiently and economically, but there appear to be opportunities to enhance the Policy and its implementation.

Further opportunities to enhance the Policy and its implementation are outlined as follows.

  1. Streamline the DAC appointment process and reduce its unpredictability.

It is necessary to reduce the excessive delays and the unpredictability of the process for obtaining ministerial approval of an external DAC member once a prospective member is selected and agreed upon by the Comptroller General and the Deputy Head. The DAC appointment process must be streamlined because a number of current appointments to DACs will be expiring soon. The renewal of existing DAC members and the appointment of new members need to be carried out quickly to avoid any vacancies in DACs, which would limit the contributions that DAC members are making. In addition, the unpredictability of the process needs to be minimized so that the most qualified DAC candidates are approved and highly qualified DAC candidates are not dissuaded from seeking a DAC membership. One option for streamlining the process and reducing its unpredictability is to delegate Treasury Board ministerial approval to the Secretary of the Treasury Board.

  1. Resolve issues with the current classification of internal auditors in the federal government and ensure that departments are able to staff their internal audit function with a broad range of skills.

One of the intended outcomes of the Policy is to increase the capacity and strengthen the professionalism of the internal audit function within departments. The evaluation provides evidence that the Policy indeed led to the development of a more professional internal audit function within departments as a result of adopting the IIA Professional Practices Framework and encouraging auditors to obtain a professional certification (e.g., a CIA designation). Training expenditures and the number of auditors who have CIA designations have both increased.

To meet the requirements of the Policy on Internal Audit and to address the decrease in the number of internal audit staff that occurred in the 1990s, more internal auditors were needed. The number of internal audit staff in the core public service was 190 in 2005 compared with 479 in 2010. This increase in demand for internal auditors has made it more difficult to attract sufficient internal audit staff. The evaluation also found that CAEs and internal audit staff believe that there is a need for a different classification for internal auditors in order to increase the professionalism of the internal audit function. In particular, CAEs and internal audit staff indicated that educational requirements and compensation levels within the AS category are factors contributing to the difficulties in attracting and retaining qualified internal audit staff.

The evaluation found that there is a shortage of qualified and experienced internal auditors in the federal government and that attracting and retaining qualified internal audit staff remains a challenge. Evidence indicated that the lack of qualified, experienced auditors negatively affected audit coverage and the quality of the work performed. To increase audit coverage and quality, the issues related to attraction and retention need to be resolved.

When addressing the issues related to attracting and retaining internal auditors, the solution must be consistent with the policy direction on professionalizing the internal audit community. In addition, the solution must take into account the concerns expressed by internal auditors and CAEs about the differences in compensation and educational requirements for the various classifications used for the internal audit function. Finally, the solution must include an HR plan that reflects the need for individuals who have a broad range of skills and experience in order to create the balance required for an effective internal audit group. HR planning is required to ensure a common understanding of the duties of the internal audit function and the core competencies needed to carry them out. This planning will support consistent work definitions for internal auditors and enable departments to better balance their HR plans to ensure that their internal audit staff have the desired combination of skills. Consistency in HR planning, work definitions and compensation for auditors will help resolve the recruitment and retention challenges noted in the evaluation.

  1. Investigate alternatives to the current method of dividing federal government departments and agencies into LDAs and SDAs.

It is necessary to determine whether there are alternatives or supplemental approaches that would better balance Policy requirements with the level of risk and the capabilities of different sizes and types of federal government departments and agencies. Although not an exhaustive list, some options proposed by respondents that appear worthy of investigation include the following:

  • Create a separate category for micro-agencies that has reduced requirements for participating in OCG horizontal audits to reflect the fact that most SDAs are very small (42 of the 50 SDAs account for only about one quarter of all SDA expenditures and approximately 0.1 per cent of all federal government expenditures). However, it is also necessary to take into consideration that even small organizations can pose a high level of risk to the federal government (e.g., negative press coverage).
  • Add the level of risk to the two existing criteria for determining whether a federal government department or agency should be considered an LDA or an SDA (i.e., $300 million in annual expenditures and 500 FTEs).
  • Align the criteria used by the Policy on Internal Audit with those used by other Treasury Board policies (e.g., Policy on Evaluation).
  • Increase the number of small LDAs that share a CAE and/or DAC, particularly where two LDAs have similar mandates.
  • Encourage some LDAs to share their CAEs and DACs with the SDAs with which they are affiliated and where there is no potential for conflict of interest.
  1. Limit the number of former public servants who are external members of DACs.

The evaluation data indicated concerns related to the proportion of former public servants who serve on DACs, particularly that having too many former public servants on DACs may reduce the potential for fresh and external perspectives. It is ideal for a DAC to have external members who have a mix of private and public sector skills. The most appropriate composition for a typical DAC that has three external members is as follows:

  • One DAC member who has public sector skills;
  • One DAC member who has skills in financial management, accounting or auditing; and
  • One DAC member who has private sector experience in a field related to the operations of the department.

Regardless of whether a DAC member is from the public or private sector, it is also important to ensure that the required skill sets and competencies are covered when selecting the most appropriate members for a particular DAC. It is preferable to have a maximum of one former public servant and a minimum of two private sector representatives serving as external members on each DAC.

  1. Provide guidance on implementing annual overview requirements.

Since the Policy was introduced, expectations have declined regarding the scope and nature of assurance that should be provided to the Deputy Head by the departmental internal audit function. Expectations have changed from the provision of a holistic opinion (2006 Policy) to an annual assurance report (2009 Policy) and most recently to an annual overview report. The OCG should investigate ways to implement annual overview requirements, and additional guidance should be provided to LDAs in this respect.

Recommendations

Because of the many positive effects of the Policy on Internal Audit and the widespread satisfaction with it, there is no impetus to make significant changes to the Policy. The following recommendations deal primarily with modifications that could be made to further increase the Policy’s effectiveness.

It is recommended that the OCG:

  1. Investigate ways to streamline the DAC appointment process and reduce its unpredictability in order to attract and retain qualified DAC members.
  2. Resolve issues regarding the current classification of internal auditors in the federal government and ensure that departments are able to acquire internal audit staff who have a broad range of skills.
  3. Examine alternatives to the current method of dividing federal government departments and agencies into LDAs and SDAs.
  4. Limit the number of former public servants who are external members of DACs to ensure that DACs are composed of members who have a broad range of skills.
  5. Investigate how annual overview requirements can be implemented in order to provide guidance to LDAs.

I. Methodology

This section describes the purpose and methodology used to perform the evaluation.

A. Purpose of Evaluation

This evaluation was undertaken as per the requirements of section 5.8.6 of the Policy on Internal Audit and as part of the Secretariat’s Five-Year Evaluation Plan. The objective of the evaluation was to assess the relevance and performance of the Policy on Internal Audit. The 2006 Policy on Internal Audit specifically states that an evaluation of the Policy is to occur and a report is to be provided to the Treasury Board by April 1, 2011. The evaluation covers the period from April 1, 2006, to March 31, 2010. The scope of the evaluation consists of the 2006 Policy on Internal Audit and adjustments made to it as of July 1, 2009.

B. Evaluation Methodology

As stipulated in the Standard on Evaluation for the Government of Canada, evaluators considered the risks associated with the Policy on Internal Audit. To address these risks, the data for the evaluation was collected in two phases. In the first phase, evaluators collected data through a review of documents and files, key informant interviews, a cost analysis, and surveys. Evaluators then analyzed the quality of the data gathered from the first phase (e.g., response rate, recall factor of respondents, key issues) to design the methods to be used to collect data in the second phase, namely, the case studies and focus groups. This strategy enabled the evaluators to adjust the data collection strategies to more effectively cover all aspects of the Policy and focus on the key issues.

As a result of the preliminary evaluation carried out in the first phase, the evaluation team identified the need to further explore the importance of the DACs, given their innovative and major role under the Policy. In addition, the preliminary evaluation identified the need to explore in more detail the effects of the differences in CAE reporting relationships, as well as the issues related to obtaining a different classification for internal auditors.

The methods used to collect data for the evaluation of the 2006 Policy on Internal Audit are outlined as follows.

1. Detailed Document and File Review

This review consisted of examining the available documents in detail, including the following:

  • The Policy and its guidelines;
  • Previous policies, evaluations and reviews of the current and previous policies;
  • CAE and DAC reports;
  • Reference and other materials produced by the OCG to facilitate implementation of the Policy;
  • Studies conducted by the OCG and other federal government organizations; and
  • Cost information compiled by the OCG on internal audit expenditures in the federal government.

2. Key Informant Interviews

A total of 25 key informant interviews were conducted with individuals involved in designing and implementing the Policy, and with experts in the field of internal audit, specifically

  • Current and previous Comptrollers General of Canada;
  • Current and previous staff of the OCG who were involved in Policy design and implementation;
  • OCG audit staff involved in horizontal departmental audits;
  • Senior managers at the Treasury Board of Canada Secretariat;
  • Representatives of internal audit associations (e.g., the IIA); and
  • Academics and experts in the field of internal audit.

3. Survey of Internal Audit Stakeholders

The following types of internal audit stakeholders were surveyed.

  1. Deputy heads: Evaluators interviewed 54 deputy heads, including respondents from both SDAs and LDAs.

    1. Deputy heads of LDAs—Of the 46 federal government departments and agencies classified as LDAs under the Policy, 29 deputy heads (about two thirds of all deputy heads) of these organizations were engaged.
    2. Deputy heads of agents of Parliament and the Public Service Commission of Canada—The Policy states that, although the principles of the Policy as it applies to LDAs will apply to the seven agents of Parliament and the Public Service Commission of Canada, the deputy heads of these organizations may authorize any departures from specific Policy requirements as they deem appropriate in light of the governance arrangements, statutory mandate and risk profile of the organization. To examine this unique group, evaluators conducted five interviews with deputy heads of agents of Parliament and the Public Service Commission of Canada. For the purposes of this evaluation, evaluators combined the responses obtained from the deputy heads of agents of Parliament and the Public Service Commission of Canada with those obtained from deputy heads of LDAs because all of these organizations must comply with similar requirements under the Policy.
    3. Deputy heads of SDAs—20 deputy heads of SDAs were interviewed (40 per cent of the total number of SDA deputy heads). The 50 SDAs in the federal government are small organizations that collectively account for less than one per cent of total federal government expenditures.
  2. CAEs—All CAEs in LDAs, agents of Parliament and the Public Service Commission of Canada were surveyed using an online questionnaire. In addition, evaluators also placed follow-up telephone calls in some cases to clarify responses to the questionnaire and to increase the response rate. Responses were received from 48 CAEs. Taking into account vacancies and situations where one CAE serves two LDAs, there are 52 CAEs in total. Therefore, the response rate was 92 per cent.
  3. DAC members—All external DAC members were surveyed using an online questionnaire. When necessary, follow-up telephone calls were made in order to clarify responses to the questionnaire and to increase the response rate. Evaluators obtained responses from a total of 97 DAC members, which corresponds to a response rate of 78 per cent.
  4. Chief financial officers (CFOs)—All CFOs in LDAs, SDAs and agents of Parliament were surveyed online, and follow-up telephone calls were placed to increase the response rate. Responses were obtained from 34 CFOs in LDAs and agents of Parliament, which corresponds to a response rate of 63 per cent. Evaluators obtained responses from 25 CFOs in SDAs, which corresponds to a response rate of 50 per cent.
  5. Internal audit staff of federal government departments and agencies—Internal auditors were surveyed using an online questionnaire, and follow-up telephone calls were placed to increase the response rate. Although evaluators initially used the membership list from the Ottawa chapter of the IIA to determine the internal auditors to be surveyed, they found the list to be out of date and incomplete. The Government of Canada employee directory was therefore used to develop a list of internal auditors for survey purposes. In addition, evaluators requested that all CAEs ask all of their internal audit staff to complete the online survey. Evaluators obtained a total of 209 completed surveys from departmental internal audit staff, which corresponds to a response rate of approximately 43 per cent.
  6. Assistant deputy ministers (ADMs)—Evaluators conducted an online survey and follow-up telephone calls were place to a sample of ADMs. To determine which ADMs were to be surveyed, evaluators requested that all CAEs provide the names of ADMs who were subject to an internal audit completed in 2009–10. A total of 133 names were provided and a sample of 75 were surveyed, yielding responses from 54 ADMs. That number represents 41 per cent of the ADMs on the list.

4. Cost Analysis of Internal Audit Expenditures

To obtain information on the costs incurred in Policy implementation, all CAEs were asked to provide information on internal audit expenditures and the number of FTEs devoted to internal audit in 2005–06 and 2009–10, as well as their estimated internal audit expenditures. Evaluators also requested a breakdown of internal audit expenditures in order to determine what was spent on internal audit staff salaries and benefits, professional services, training, and DAC costs.

5. Case Studies

Evaluators conducted six case studies. The topics of the case studies were selected after a large portion of the key informant interviews and stakeholder surveys were completed to obtain a better understanding of the key issues that should be investigated. The topics for the six case studies are outlined as follows.

  1. Case studies 1, 2 and 3: The purpose of these studies was to determine whether there were any differences in contributions made by DACs, based on who was chair of the DAC. The following three alternative DAC models were the subject of a separate case study:
    • The deputy head is the DAC chair.
    • An external member is the DAC chair.
    • The deputy head does not attend DAC meetings but is briefed on them.
  2. Case study 4: This study examined the issues related to obtaining a different classification for internal auditors. Funding was provided to the OCG to help reconfigure the internal audit community (including a separate classification for internal auditors in the federal government), but a separate classification has not yet been created. Based on the pretest results, a different classification was considered to be an important part of enhancing the professionalism of the internal audit community. The purpose of the case study was to examine the constraints that have prevented the completion of the proposed reconfiguration of the internal audit community and the likely success of current efforts to accomplish that goal.
  3. Case studies 5 and 6: The purpose of these two studies was to analyze different CAE reporting relationships. According to MAF data on CAE reporting relationships, 27 of 45 CAEs report "solely and exclusively to the deputy head," and 18 of 45 CAEs report "solely and substantively to the deputy head." Evaluators conducted two case studies to examine the effects of different CAE reporting relationships, one where CAE reports "solely and exclusively to the deputy head" and one where the CAE reports "solely and substantively to the deputy head."

6. Focus Groups

Evaluators conducted four focus groups, prepared guides for each of the groups, and submitted them to the Evaluation Working Group for feedback. Based on the feedback obtained, evaluators finalized the guides and sent them to participants a few days prior to the session. The four focus groups are described as follows.

  1. Deputy head focus group: The purpose of this group was to obtain elaboration on issues identified during the survey of deputy heads and to address the case study topics. Four deputy heads participated in the focus group.
  2. CAE focus group: The purpose of this group was to obtain elaboration on issues identified during the survey of CAEs and to address the case study topics. The criteria used to select the focus group participants included the CAE reporting relationship (i.e., "strong" and "acceptable" departmental MAF ratings); DAC chair (i.e., external chair, deputy head chair); sector (e.g., economic operations, government operations); and size of organization (FTEs). Based on these criteria, 18 CAEs were initially selected and invited to participate in the focus group. Several CAEs were unable to attend because of other activities, and a few declined at the last minute. Six CAEs attended the focus group. The CAEs who attended were representative of the CAE population, as they were from different types and sizes of departments, and had different reporting relationships and different DAC chairs.
  3. External DAC member focus group: The purpose of this group was to obtain elaboration on issues identified during the survey of external DAC members and to address the case study topics. The criteria employed to select the focus group participants included the CAE reporting relationship (i.e., "strong" and "acceptable" departmental MAF ratings); DAC chair (i.e., external chair, DH chair); sector (e.g., economic operations, government operations); and size of organization (FTEs). A total of 14 DAC members were invited to attend the focus group. Five members attended either in person or by telephone. The DAC members who attended were representative of the DAC population, as they were from different types and sizes of departments, and had different CAE reporting relationships and different DAC chairs.
  4. Departmental internal audit staff focus group: The purpose of this group was to obtain elaboration on issues identified during the survey of internal audit staff and to address the case study topics. The criteria employed to select the focus group participants included years of internal audit experience; CAE reporting relationship (i.e., "strong" and "acceptable" departmental MAF ratings); DAC chair (i.e., external chair, DH chair); sector (e.g., economic operations, government operations); and size of organization (FTEs). A total of 16 internal audit staff were invited to the focus group, eight of whom attended. The staff who attended were representative of the internal audit community.

7. Review of Similar Policies in Other Jurisdictions

Evaluators conducted a comparative analysis of the Policy with similar policies in other jurisdictions. The other jurisdictions investigated were the United Kingdom, Australia and Ontario. In addition, extensive Internet and library research was conducted to obtain available information on the internal audit policies and practices in these jurisdictions. Evaluators then conducted telephone interviews with representatives of federal government internal audit departments and branches in these jurisdictions.

C. Limitations

The key limitations are outlined as follows.

  1. Because of the small number of informants interviewed, a detailed breakdown of findings by subgroup could not be completed.
  2. Because the Policy on Internal Audit is complementary to a number of other policies and initiatives that focus directly on areas such as internal control and risk management (e.g., Policy on Internal Control, Risk Management Policy, Federal Accountability Act and Policy Suite Renewal), it is difficult to attribute the impact of the Policy on Internal Audit because its intended outcomes are shared with complementary programs.
  3. Because some departments are at various stages of implementing the Policy on Internal Audit, it is difficult to generalize findings across departments.
  4. Given that only a small proportion of the individuals invited chose to attend the focus groups, the observations of the focus groups cannot be generalized for the entire population.

D. Report Outline

The next section provides a brief description of the 2006 Policy on Internal Audit, including its objectives and intended outcomes. Section III provides the evaluation findings and conclusions for each evaluation issue. The last section summarizes the key evaluation conclusions and provides recommendations to enhance the Policy’s effectiveness.

II. Description of the 2006 Policy on Internal Audit

This section provides a brief profile of the 2006 Policy on Internal Audit.

A. Rationale

Many factors internal and external to the federal government influenced the development of the 2006 Policy on Internal Audit. The Auditor General of Canada played an important diagnostic and influential role by auditing the federal government internal audit function in 1994 and 2004. In the 2004 November Report of the Auditor General of Canada, Chapter 1, “Internal Audit in Departments and Agencies,” the Auditor General noted that considerable work remained to strengthen the internal audit function in the Government of Canada. The Auditor General also mentioned that, despite additional funding for internal audit in the past four years, the same problems have remained for more than a decade. Some of the other internal and external factors that influenced the development of the 2006 Policy on Internal Audit include the following:

  • Public scrutiny increased as a result of the Enron, Sarbanes-Oxley and Gomery scandals;
  • The Public Accounts Committee indicated that internal audit lacked sufficient independence from line management.
  • Audit committees were not independent.
  • Deputy heads were not consistently provided with the independent assurance they need to support them in discharging their responsibilities.
  • Internal audit was not focused on assurance; its focus on reviews and consulting presented a potential conflict of interest.
  • There was a lack of consistency in internal audit capacity, skills and practice throughout the federal environment.
  • Internal auditors were not mobile within government.
  • The government was unable to articulate how well its control framework was working within an entity or across government.
  • The audit environment became more complex and driven by information technology.
  • The provisions of the Access to Information Act weakened the audit regime.

B. Policy Description

In October 2005, the Comptroller General submitted to the Treasury Board a proposed new Policy on Internal Audit. The Policy was approved and took effect on April 1, 2006, along with three related directives. Implementation occurred in stages, between the effective date and April 1, 2009. The new Policy addressed the following:

  • The government would sustain a strong, credible internal audit regime that holds the confidence of the government and contributes directly to effective risk management, sound resource stewardship and good governance.
  • The government would ensure the independence of the internal audit function from line management by requiring DACs to have a majority of external members recruited from outside the federal public administration and by requiring CAEs to report directly to the deputy head.
  • CAEs would provide deputy heads with annual opinions on the adequacy of risk management, control and governance processes, and would report on individual risk-based audits.
  • The OCG would take on new responsibility for horizontal and sectoral audits, as well as focused horizontal or sectoral auditing in SDAs.
  • The OCG would establish an audit committee to provide advice and guidance on its work in SDAs.
  • The OCG would report annually to the Treasury Board on the state of risk management, control and governance processes across government, and would report on audit work within departments.
  • The OCG would establish standards and guidance that align with internationally recognized internal audit practices.
  • The OCG would conduct horizontal audits integrated with departmental audit activity, and would provide advice and services on internal audit matters.
  • The OCG would implement performance monitoring and accountability reporting for internal audits.
  • The Policy created a framework to better balance the internal audit responsibilities of deputy heads and the OCG.
  • The Policy stipulated an investment in the recruitment of skilled professionals and in the professional development of auditors.
  • The Policy reinforced the assurance role of internal audit in accountability relationships, decision making and program improvement, and expanded the role as it pertains to risk management, control and governance.
  • The Policy stipulated a comprehensive, government-wide approach to the way internal audit activities are planned and carried out in federal departments.

C. Objectives

The objective of the 2006 Policy on Internal Audit is to strengthen public sector accountability, risk management, resource stewardship and good governance by reorganizing and bolstering internal audit services across government. In 2009, the Policy objective was updated to state the following: “The objective of the Policy is to support strong and accountable public sector management by ensuring effective internal auditing within departments and across governments.”

D. Related Legislation

The Federal Accountability Act was enacted in December 2006. It is a key framework, driver and enabler for the government’s public accountability agenda. Within this omnibus legislation, several important steps were taken that pertain directly to the internal audit function. The Act brought the first recognition of internal audit in Canadian federal legislation. The Act included a specific requirement for departments to have an appropriate internal audit capacity and to have a DAC. It also designated deputy heads as the accounting officers of their respective departments, answerable to parliamentary committees and responsible for areas such as systems of internal control and compliance. In addition, changes to the Access to Information Act allowed for a discretionary exemption for internal audit working papers for up to 15 years.

A more recent development that further enhanced public accountability was the preparation of the financial management policy suite for the federal government, including the Policy on Financial Management Governance and the Policy on Internal Control. The latter focuses primarily on internal controls over financial reporting and letters of representation from deputy heads and CFOs.

E. Policy Implementation

The 2006 Policy on Internal Audit applies to “departments” within the meaning of section 2 of the Financial Administration Act. This does not include the Canada Revenue Agency or Crown corporations. Of the 104 departments covered by the Policy, 46 are LDAs (meaning that they have annual expenditures of $300 million or more, or 500 or more FTEs). There are 50 SDAs (less than $300 million in annual expenditures and fewer than 500 FTEs). With regard to the remaining federal government departments, the Policy states that the principles of the Policy, as it applies to LDAs, will apply to the offices of agents of Parliament and to the Public Service Commission of Canada. In that respect, the deputy heads of these organizations may authorize any departures from specific policy requirements as they may deem appropriate in light of the governance arrangements, statutory mandate and risk profile of the organization.

Some of the key requirements and provisions of the 2006 Policy on Internal Audit that pertain to LDAs include the following:

  • CAEs must report directly to the deputy head.
  • DACs must have a majority of external members recruited from outside the federal public administration.
  • A risk-based internal audit plan (i.e., a plan that addresses the areas of higher risk and significance) must be developed and implemented.
  • An internal audit capacity must be provided that is appropriate to the needs of the department and that operates in accordance with the Policy and professional internal audit standards.
  • Chief audit executives must provide deputy heads with added assurance on the adequacy and effectiveness of risk management, internal control and governance processes within the department.

To help implement the Policy, accompanying funding was provided to enable LDAs to carry out activities such as hiring and training internal audit staff, and to pay for the costs of external DAC members.

Some of the key requirements and provisions of the 2006 Policy on Internal Audit that pertain to SDAs include the following:

  • The OCG will conduct horizontal and other audits of SDAs each year and will provide deputy heads with copies of all relevant audit reports.
  • The Comptroller General is responsible for establishing a Small Departments and Agencies Audit Committee (SDAAC) to provide a review, advice and recommendations on internal audits of SDAs conducted by the OCG.
  • The OCG will facilitate access to independent and qualified internal audit resources when deputy heads of SDAs determine a need for internal audit work beyond that performed by the OCG.

In addition to these requirements and provisions, the Policy states that the Comptroller General is responsible for focused, sustained functional leadership of internal audit across government in order to build and develop capacity, ensure adequate levels of professionally qualified resources, and ensure adherence to professional standards and rigour in the delivery of internal audits.

The implementation of the Policy is a mixed model; it is decentralized to departments, with strong, centralized functional guidance and operational capacity at the OCG. The Policy outlines the responsibilities of the OCG, as well as how departmental and OCG roles and responsibilities are expected to interact.

F. Interim Assessment of the Policy

In 2008, the OCG performed an interim assessment of the implementation of the 2006 Policy on Internal Audit. The objective was to identify critical policy changes that were required immediately. The assessment criteria were the Policy’s relevance, alignment with statutory amendments brought about through the Financial Administration Act, criticality, practicality and achievability, and harmonization with the reporting requirements set out in federal government financial management policies. In 2008, the OCG reviewed prior assessments, conducted a line-by-line review of the Policy as well as the Policy’s directives and guidelines, and sought feedback from stakeholders. The outcome of the interim assessment resulted in several amendments to the Policy, effective July 1, 2009, that codified the practice and understanding of roles and functions that were operative early on in the life of the Policy. The 2009 amendments were not new, but were based on practice and experience as well as legislative changes that occurred in 2006–07. Some of the key amendments made to the Policy in 2009 included the following:

  • Updated the language subsequent to the 2007 changes to the Financial Administration Act concerning the role of the deputy head as the accounting officer and the advisory role of the DAC.
  • Reframed the role of CAEs in providing assurance on departmental governance, risk management and control processes.
  • Harmonized the OCG’s reporting role and the delegation of authority to the President of the Treasury Board with requirements delineated in the financial management policy suite.
  • Permitted agents of Parliament and the Public Service Commission of Canada to authorize specific risk-based departures from the Policy, insofar as the underlying Policy principles are adhered to, and the legislative requirements applicable to internal audit are observed.
  • Removed the concept of the DAC as a direct assurance provider and instead stated that DACs should provide the Deputy Head with advice and recommendations regarding the sufficiency, quality and results of assurances.
  • Highlighted the notion of risk-targeted (versus comprehensive) audit coverage and sought to inject risk considerations to reduce the process burden on DACs.
  • Adjusted the requirement for DAC comments on Departmental Performance Reports.
  • Removed the need for CAEs to provide annual holistic opinions to deputy heads and DACs on the effectiveness and adequacy of risk management, control and governance processes in their departments. The revised Policy states that the CAE should provide the Deputy Head with an independent assurance report on the adequacy and effectiveness of risk management, control and governance processes within the department.
  • Changed the timing of reports to the Treasury Board from periodic to annual.
  • Removed the notion of in camera meetings of DACs with their respective minister.

G. Resources

In keeping with one of the cornerstone objectives of the 2006 Policy on Internal Audit, the Treasury Board approved the progressive allocation of incremental resources to help the internal audit community implement the Policy’s new requirements. This began with incremental investments of approximately $13 million in 2006–07 and $24 million in 2007–08. Once the strategy for reconfiguring the internal audit community is fully implemented, total incremental funding across government is expected to rise to about $40 million on an ongoing basis. The funding approved by the Treasury Board is incremental to investments made by departments.

In addition to this allocated funding, the OCG established an operational capacity to conduct horizontal audit work across SDAs and LDAs. In mid-2008, the OCG also established an omnibus supply arrangement for audit and related support services (called Professional Audit Support Services, or PASS). PASS provides departments and agencies with timely access to qualified, contracted audit services. This arrangement is important as an essential supplement to internal audit capacity during the implementation transition period and beyond.

H. Activities

The following is a brief description of some activities that have been carried out to date as a result of the Policy on Internal Audit.

  1. Independent DACs: Working with departments and the internal audit community, the OCG put in place processes to recruit, select, retain and compensate external members of the independent DACs, which are mandated to provide deputy heads with advice on the functioning and adequacy of departmental governance, risk management and control frameworks and processes. Recruitment activities have resulted in the establishment of a DAC for almost every LDA (47 audit committees with 152 external members). Some activities carried out by DACs include preparing an annual report for the Deputy Head, and providing advice and recommendations as requested by the Deputy Head on emerging priorities, concerns, risks, opportunities, and accountability reporting.

    To support the evolution of an effective and sustainable community of practice for DACs, the OCG implemented a curriculum for learning and engagement. The curriculum includes an orientation to the federal government and core subject-matter workshops (e.g., risk management). A generic DAC Charter was developed in 2006, and the terms and conditions of appointment as well as the Guidebook for a Departmental Audit Committee were released in 2007. The latter provides reference tools to help DAC members fulfill their role and responsibilities.

    The Government of Canada Audit Committee was established to provide advice to the Secretary (Deputy Head) of the Treasury Board of Canada Secretariat and to provide oversight support to the Secretary and the Comptroller General in relation to the government wide activities of the OCG regarding its functional leadership, operations, monitoring and reporting responsibilities in the area of internal audit. To support the OCG’s horizontal audit responsibilities in relation to SDAs (as called for in the Policy), the SDAAC was also established. The SDAAC performs a role similar to that of a DAC.

  2. CAE appointments: With one exception, all LDAs in the federal government have appointed CAEs that report directly to the Deputy Head. The CAEs were hired by the department in consultation with the OCG. CAEs are responsible for leading the internal audit function within the department, including establishing three-year, risk-based audit plans and performing risk based internal audits as needed to provide the Deputy Head with an independent annual overview report on the adequacy and effectiveness of risk management, control and governance processes within the department. In addition, the CAE is responsible for following up on management action plans to address any issues raised as a result of the internal audits. The OCG has established a training and mentoring program for CAEs.

  3. Internal audit standards and guidance: As part of the Policy’s implementation, Internal Auditing Standards for the Government of Canada were produced to prescribe the internal auditing standards that are to be applied in all departments subject to the Policy on Internal Audit. The standards stipulate that the Government of Canada has adopted the IIA Professional Practices Framework and that all government departments must meet IIA standards in fulfilling their internal audit responsibilities, unless the standards conflict with the Policy or any related directives or guidelines provided by the Comptroller General or the Treasury Board. By establishing minimum standards, the Policy is intended to ensure that all departments have common internal audit standards and processes; however, this does not mean that departments need to employ identical internal audit processes.

    In fall 2006, the OCG prepared a draft maturity model to set out expectations in relation to the Policy and to help departments assess their current internal audit operation, identify priorities and develop a plan of action. At the end of 2006, a draft internal audit Policy, standards and a practices manual were issued by the OCG to the federal internal audit community. As a result of further analyses and comments received from CAEs, this professional practices product was revised and resulted in the creation of the OCG Internal Audit Reference Centre, which incorporates the IIA Professional Practices Forum and provides a fully integrated, single-window, online reference library to guide audit practitioners in their work. The Web-based beta version of the reference centre was launched at the beginning of 2009 and has since been refined to reflect subsequent developments (e.g., the 2009 IIA Professional Practices Framework and amendments to the Policy in 2009).

    The development of a core controls framework facilitated the conduct of audits and gave assurance to CAEs. The OCG developed and released a draft framework on core management controls in 2007, drawing from landmark initiatives such as the Control Objectives for Information and related Technology (COBIT), Criteria of Control Board (CoCo), and the Committee of Sponsoring Organizations of the Treadway Commission (COSO), and organized around the 10 key elements of MAF.

    A further initiative involves developing a risk assessment methodology to support internal audit planning, specifically by creating a government-wide internal audit plan at the whole of government level and risk-based audit plans at the departmental level. Initial guidance on developing a risk assessment methodology was released as part of the OCG Internal Audit Reference Centre at the beginning of 2009.

    The OCG has begun development work on practice inspections of the internal audit operations in individual departments and agencies. This development work, as well as related work on the OCG Internal Audit Maturity Model and the OCG Internal Audit Reference Centre, has yielded a practice inspection suite (e.g., a manual, a guidebook and a self-diagnosis) that helps departments and agencies participate in this activity and guides those conducting practice inspections. The most recent guidance is the Internal Audit Practice Inspection Guidebook, which was released in June 2010.

  4. Horizontal and sectoral audits: The OCG produced an initial 2007–10 horizontal audit plan using a risk assessment methodology. Along with this plan, the OCG has completed a number of horizontal audits of LDAs and SDAs.
  5. HR strategy and staffing: Since the Policy was introduced, OCG Capacity Building and Community Development has worked with the federal government internal audit community to build an HR framework for internal audit. The HR framework has several components, including a human capital plan; standardized organizational models and pre classified work descriptions; competency profiles for these work descriptions; resourcing to ensure that recruitment and capacity building activities meet the needs of the community; learning, professional and management development; and community outreach.
  6. Internal auditor training: The last few years have seen enhancements to the training curriculum for internal auditors. The core of this curriculum, which has been in place for several years, includes the following elements: orientation to internal audit, conducting internal audits, and managing internal audits. Courses have been developed and delivered with the Canada School of Public Service, including training in risk-based audit planning, assessment of internal control, communications, and ethics. Periodic armchair sessions also provide perspectives on specific aspects of the Policy and its implementation. In addition, the OCG is working with the IIA to provide training to support internal audit certification.
  7. Accountability of the internal audit function: As part of MAF, the Treasury Board of Canada Secretariat performs an annual assessment of the internal audit function in federal government departments. The MAF assessment examines the progress made by the internal audit function within federal government departments
  8. Annual assurance reporting: One of the amendments to the 2006 Policy, effective July 1, 2009, stipulated that, instead of providing a holistic opinion, CAEs should provide the Deputy Head with an independent assurance report on the adequacy and effectiveness of risk management, control and governance processes within the department. Upon further consideration of this requirement, the OCG’s current vision is for the CAE to provide the Deputy Head with an annual overview report on the audit and other work performed. The annual overview report should give the Deputy Head a timely overview of the internal audit findings resulting from the execution of the risk-based audit plan, summarized in the context, mandate, priorities and risk profiles of the organization. Draft OCG guidance on the annual overview reporting requirements is in progress.

I. Intended Outcomes

The intended outcomes of the 2006 Policy on Internal Audit are outlined as follows.

  • Immediate Outcomes
    • Independence of the internal audit function within departments;
    • Increased assurance and advice to deputy heads regarding risk management, control and governance;
    • Increased capacity and strengthened professionalism of the internal audit function within departments; and
    • Audit coverage appropriate to the level of risk.
  • Intermediate Outcomes
    • Increased deputy head confidence in the assurance and advice provided by the internal audit function and DACs on risk management, control and governance;
    • Increased effectiveness of the internal audit function;
    • Increased management action on internal audit recommendations leading to improved risk; and management, governance and internal control in audited areas.
  • Long-Term Outcomes
    • Support for deputy heads in their role as accounting officers;
    • Strong credible internal audit regime;
    • Improved overall departmental risk management, internal control, governance and resource stewardship; and
    • Increased ability of departments and agencies to effectively meet their objectives.
  • Ultimate Outcome
    • Strong and accountable public sector management.

J. Logic Model

Figure 1 provides a logic model for the 2006 Policy on Internal Audit and depicts the causal relationships and links between the input, activities, outputs and intended outcomes of the Policy. The links between the activities and the hierarchy of outcomes in the logic model constitute testable hypotheses, which form the essence of the evaluation of the Policy. The intended outcomes are grouped into immediate, intermediate, long-term and ultimate categories. All of the inputs, activities and outputs shown in the logic model contribute to the ultimate outcome of the Policy, which is “strong and accountable public sector management.” The evaluation of the Policy concentrates primarily on assessing whether the immediate and intermediate outcomes have been achieved, given that they are more quantifiable and directly attributable to activities carried out as a result of the Policy. In addition, sufficient time may not have elapsed to measure whether all of the long-term outcomes (as well as the ultimate outcome) of the Policy have been achieved.

Figure 1. The logic model for the 2006 policy on internal audit
Figure 1. The logic model for the 2006 policy on internal audit. Text version below:
Figure 1. The logic model for the 2006 policy on internal audit - Text version

This graphic image provides the logic model for the 2006 Policy on Internal Audit.

As illustrated, the logic model shows that the ultimate outcome of the Policy is strong and accountable public sector management.

The four long-term outcomes are that Deputy Heads are supported in their roles as accounting officer; strong credible internal audit regime; increased ability of departments and agencies to effectively and efficiently meet their objectives; and improved overall department risk management, control, governance, and resource stewardship. The three intermediate outcomes presented are Deputy Heads have increased confidence in the assurance and advice provided by the internal audit function and DACs on risk management, control, and governance; increased effectiveness on the internal audit function; and increased management action on internal audit recommendations leading to improved risk management, governance, and internal control to audited areas. Four immediate outcomes are provided: independence of the internal audit function within departments; increased assurance and advice to Deputy Heads regarding risk management, control, and governance; increased capacity and strengthened professionalism of the internal audit function within departments; and audit coverage appropriate to level of risk. The following outputs lead to the stated outcomes: Chief Audit Executive support to DAC and Deputy Head, Chief Audit Executive Annual Reports, DAC advice on quality of assurance provided and adequacy of risk management, governance, and internal control frameworks and processes, DAC reports, Policy guidance documents and related directives, enhanced internal audit standards used by all departments, new internal audit tools and work processes, training and development, ongoing assessment of the state of the internal audit function across government, internal audit function quality assurance reviews and internal/external assessments, sustainable capacity appropriate to the internal audit needs of the department, enhanced internal auditor competencies, internal auditor certification, Deputy Head approved 3-year Risk-based Audit plans, Office of the Comptroller General approved horizontal and sectoral audit plans, audit intelligence, departmental audit reports provide assurance on governance, risk management, and internal control in high risk areas, horizontal and sectoral audits, follow-up audits of management action plans, and the Comptroller General’s report to Treasury Board. The activities leading to the production of the named outputs are as follows: appointing qualified Chief Audit Executive who reports to the Deputy Head; Establishing an independent Departmental Audit Committee, Departmental Audit Committee training; Office of the Comptroller General guidance and leadership; Monitoring policy implementation and practice inspections; Staffing, Human Resources strategy; Training and development of internal auditors; Annual risk based planning; and Conducting internal audits. To implement the activities that produce the outputs which result in the stated outcomes, the following inputs have been allocated: Total TBS funding of $72.5 million over three years. $60.2 million dollars over three years to departments and agencies and $12.3 million /3 years to the Office of the Comptroller General.

III. Evaluation Findings and Conclusions

The 2009 Policy on Evaluation stipulates that, in the Government of Canada, evaluation is the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance and performance, and alternative ways to deliver them or to achieve the same results. Pursuant to the Policy on Evaluation, this section provides the evaluation findings and conclusions related to evaluation questions dealing with program relevance, performance and alternatives.

A. Relevance

This section presents findings and conclusions on evaluation questions dealing with the extent of the need for and the relevance of the Policy on Internal Audit.

Evaluation Question 1: Is there a continued need for the current Policy?

To assess need, a survey of stakeholders was conducted. They were asked to indicate to what extent there is a need for the current Policy on Internal Audit (i.e., the 2006 Policy with 2009 modifications), on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. Based on average ratings provided, all respondent groups indicated that the need for the current Policy is still significant, as shown in Figure 2.

Figure 2. Need for the Policy
Figure 2. Need for the Policy. Text version below:
Figure 2. Need for the Policy - Text version

This bar graph illustrates interviewed respondents’ views regarding the need for the Policy on Internal Audit. Individual groups provided the following results: Deputy Heads of Small Departments and Agencies, 4.0, Chief Financial Officers of Small Departments and Agencies, 4.0, Assistant Deputy Ministers, 4.1, Chief Financial Officers of Large Departments and Agencies, 4.3, Internal Audit staff, 4.4, Deputy Heads of Large Departments and Agencies, 4.4, Chief Audit Executives, 4.5, Departmental Audit Committee Members, 4.5 and key informants, 4.6.

The most frequent reasons given by stakeholders, particularly LDA deputy heads, for their ratings were as follows:

  • The need for independent audit advice has never been greater because of increased public scrutiny and expectations regarding accountability as stipulated in the Federal Accountability Act.
  • The Policy provides assurance for deputy heads to help fulfill their role as the accounting officers of their organization.
  • The Policy provides central guidance on the responsibilities of all stakeholders and ensures that the internal audit function is adequately resourced.
  • The Policy provides deputy heads with independent opinions to help them monitor and manage their organization.
  • The Policy supports strong and accountable management across government.

A review of the available documentation, including CAE and DAC reports, indicated that the following activities performed as part of Policy implementation have addressed most of the factors that led to the Policy’s introduction:

  • Establishment of DACS that have a majority of external members;
  • Establishment of a CAE position that reports directly to the Deputy Head;
  • Development and use of Government of Canada internal audit standards by all federal government departments and agencies;
  • Focus on assurance services;
  • Horizontal audits conducted by the OCG to ensure appropriate internal audit coverage and capacity in small entities;
  • Changes made to the Access to Information Act (as part of the Federal Accountability Act) to protect internal audit working papers for a period of up to 15 years; and
  • HR strategy developed by the OCG for the internal audit function in the federal government.

Almost all the key informants surveyed indicated that most of the factors supporting the need for the current Policy on Internal Audit have not changed. These respondents indicated that it is critical for the internal audit function to remain independent and for the capacity and professionalism of the internal audit function not to revert to the situation that existed before the Policy was introduced. Although there have not recently been any high-profile breakdowns of control in the federal government, several stakeholders indicated that the importance of a risk-based internal audit function has increased as a result of the Federal Accountability Act and the new role of deputy heads as the accounting officers of their respective organizations. In addition, respondents indicated that there is a high level of public scrutiny and expectation with regard to the accountability of federal government ministers and employees.

Conclusion: The need for the Policy remains.

Evaluation Question 2: To what extent have the internal and external factors that led to the introduction of the 2006 Policy on Internal Audit changed or remained the same?

According to the documentation reviewed, the Auditor General of Canada played an important diagnostic and influential role by auditing the federal government internal audit function in 1994 and 2004. In the 2004 November Report of the Auditor General of Canada, Chapter 1, “Internal Audit in Departments and Agencies,” the Auditor General noted that considerable work remained to strengthen the internal audit function in the Government of Canada. The Auditor General also mentioned that, despite additional funding for internal audit in the past four years, the same problems have remained for more than a decade. The report identified the following factors that, if implemented, could positively affect the quality of internal audit across government:

  • A consistent understanding by senior management of the role that internal audit can and should play;
  • A DAC that has external members who are independent from management;
  • A clear HR strategy at the departmental, central agency and government wide levels that sets out the qualifications and the appropriate number of staff for the internal audit community;
  • A focus on assurance services; and
  • A strategy to ensure appropriate internal audit coverage and capacity in small entities.

Some other factors that influenced the development of the 2006 Policy on Internal Audit include the following:

  • Public scrutiny of accountability in the government and the private sector increased as a result of Enron, Sarbanes-Oxley and Gomery scandals.
  • The Public Accounts Committee indicated that internal audit lacked sufficient independence from line management.
  • There was a lack of consistency in internal audit capacity, skills and practice throughout the federal environment.
  • Internal audit methods and standards were not applied consistently.
  • Internal auditors were not mobile within government.
  • Internal audit was not focused on assurance, and its focus on reviews and consulting presented a potential conflict of interest.
  • Audit committees were not independent.
  • The provisions of the Access to Information Act weakened the audit regime.
  • Deputy heads were not consistently provided with the independent assurance they required to help them discharge their responsibilities.

Conclusion: Most of the internal and external factors that led to the introduction of the 2006 Policy on Internal Audit have remained the same.

Evaluation Question 3: To what extent are the Policy objectives still relevant?

The Policy objectives are to strengthen public sector accountability, risk management, resource stewardship and good governance by reorganizing and bolstering internal audit across government. As evidence that the Policy objectives are still relevant, the Federal Accountability Action Plan states that independent, objective and timely internal audit services within departments are necessary because they provide assurance to deputy ministers and reinforce good stewardship practices and sound decision making. In addition, the Federal Accountability Act supports the Policy by requiring that deputy heads ensure an appropriate internal audit capacity and establish DACs.

All key informants who responded to this evaluation question stated that the Policy objectives are relevant because a professional and independent internal audit function will always be essential to public sector accountability, risk management, resource stewardship and good governance. They also indicated that, by focusing on a professional and independent internal audit function, the Policy requires federal government departments and agencies to address the areas of highest risk and to obtain assurance that the necessary controls are in place and operating effectively. These controls are critical to public sector accountability, risk management, resource stewardship and good governance.

Stakeholders were asked to indicate their overall degree of satisfaction with the Policy, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is very satisfied. If the degree of satisfaction is high, it can be presumed that the Policy objectives are still relevant. A low degree of satisfaction indicates that the Policy objectives may not be relevant; however, it could also indicate that other factors, such as Policy implementation, have not proceeded as intended. As indicated in Figure 3, there was widespread satisfaction with the Policy, as evidenced by the fact that the average rating provided by all respondent groups is 4 out of 5, and there are no significant variations in the average ratings of the different respondent groups.

Figure 3. Degree of Satisfaction With the Policy
Figure 3. Degree of Satisfaction With the Policy. Text version below:
Figure 3. Degree of Satisfaction With the Policy - Text version

This bar graph shows the various respondent groups’ levels of satisfaction with the Policy on Internal Audit. The following results are shown in graphic form: Departmental Audit Committee Members and key informants, 4.2, Chief Audit Executives, 4.1, Deputy Heads of large departments and agencies, 4.0, Chief Financial Officers of large departments and agencies, 4.0, Internal Audit staff, 3.9, Deputy Heads of small departments and agencies, 3.8, Assistant Deputy Ministers, 3.8, and finally, Chief Financial Officers of small departments and agencies, 3.8.

Of the 29 LDA deputy heads who responded to this evaluation question, 25 of them, or 83 per cent, stated that they were either satisfied or very satisfied with the Policy. The highest average satisfaction rating was provided by DAC members (4.2 out of 5); the lowest average satisfaction rating (3.8) was provided by SDA respondents and ADMs.

The most frequent reasons given by respondents for their degree of satisfaction with the Policy are as follows:

  • The establishment of DACs that have external members has been one of the most important activities carried out to achieve Policy objectives.
  • The Policy enables federal government departments and agencies to address their areas of highest risk internally, rather than relying on external assessments.
  • The Policy provides assurance to accounting officers that controls are in place to ensure public sector accountability, risk management, resource stewardship and good governance.
  • The provision of incremental funding in conjunction with the Policy has enabled the establishment of a professional and independent internal audit function within the federal government.

Although SDA deputy heads and CFOs stated that they appreciate the audit coverage provided to SDAs as a result of OCG horizontal audits, the most frequent reason given for their lower degree of satisfaction is their desire for the Policy to recognize the limited capabilities of the smaller SDAs (i.e., micro-agencies) in terms of participating in horizontal audits.

Conclusion: The Policy objectives are still relevant.

Evaluation Question 4: To what extent does the Policy meet the Government of Canada’s policy priorities, specifically those pertaining to accountability, transparency, risk management, control and governance?

The policy priorities of the Government of Canada with regard to accountability, transparency, risk management, control and governance are stipulated in the Financial Administration Act, which was introduced on April 11, 2006, to make government more accountable. The Financial Administration Act designates deputy ministers and deputy heads as accounting officers who are accountable to the appropriate committee of Parliament to answer questions related to their responsibilities. These responsibilities consist of the following:

  • Ensuring that resources are organized to deliver departmental objectives in compliance with government policies and procedures;
  • Ensuring that effective systems of internal control are in place;
  • Signing departmental accounts; and
  • Performing other specific duties assigned by law or regulation in relation to the administration of the department.

The Financial Administration Act states that independent, objective and timely internal audit services within departments are necessary to provide assurance to deputy ministers and to reinforce good stewardship practices and sound decision making. The Financial Administration Act specifies that deputy heads must ensure an appropriate internal audit capacity and establish DACs. The Policy on Internal Audit closely aligns with this Act because the Policy states that departments must have a DAC that has a majority of external members, and accompanying funding was provided with the Policy to ensure that departments have an appropriate internal audit capacity. In addition, the Policy objectives align with the Government of Canada priorities of increased accountability, transparency, risk management, control and governance, as set out in the Financial Administration Act.

Government of Canada priorities regarding internal control are also described in the financial management policy suite for the federal government, including the Policy on Financial Management Governance and the Policy on Internal Control. The Policy on Internal Audit is complementary to both of these because its intended outcomes include improved internal control and governance. A review of the Policy on Internal Control indicates that it focuses primarily on internal controls over financial reporting and letters of representation from deputy heads and chief financial officers—matters relevant to the practice of internal audit and to related governance mechanisms and processes.

All of the key informants who responded to this evaluation question stated that the objectives of the Policy on Internal Audit align with the policy priorities of the Government of Canada. These respondents stated that the Policy’s objectives to reorganize and bolster internal audit across government contribute to the Government of Canada’s policy priorities related to public sector accountability, risk management, resource stewardship and good governance.

Conclusion: The Policy aligns with the Government of Canada’s policy priorities regarding accountability, transparency, risk management, control and governance.

B. Performance: Achievement of Outcomes

This section presents the findings and conclusions regarding the extent to which the ¨Policy has achieved its intended outcomes.

Evaluation Question 5: To what extent has the Policy contributed to increasing the independence of the internal audit function from line management (exclusive of the deputy heads) across government?

The Policy states that deputy heads are responsible for appointing a qualified CAE at a senior executive level, reporting directly to the Deputy Head, to lead and direct the internal audit function. Based on a survey of CAEs and a review of CAE reports, all 46 of the large federal government departments designated as LDAs under the Policy have established a CAE position. In one case, one CAE is responsible for two organizations (the Natural Sciences and Engineering Research Council, and the Social Sciences and Humanities Research Council). Approximately 94 per cent (45 of 48) of the CAEs surveyed from LDAs, agents of Parliament and the Public Service Commission of Canada, stated the following:

  • They report directly to the Deputy Head of their organization in all aspects of their work.
  • Two of the CAEs indicated that they report functionally to the Deputy Head but administratively to another senior manager in the department.
  • One CAE indicated that he or she does not report directly to the Deputy Head, but to another senior manager in the organization.

These findings differ from the 2009 MAF ratings, which indicated that CAEs reported “solely and exclusively” to the Deputy Head in 64 per cent of the departments or agencies rated, whereas CAEs in the remaining 36 per cent reported “solely and substantively” to the Deputy Head. The primary difference between these two ratings is that a CAE reporting relationship classified by MAF as “solely and exclusively” refers to substantive matters of importance related to audit work (i.e., functional), whereas the CAE reporting relationship classified by MAF as “solely and substantively” refers to situations where the CAE reports to the Deputy Head functionally but reports to another senior manager administratively.

DAC members and CAEs were asked to indicate to what extent the internal audit function is independent from line management (exclusive of the Deputy Head) in CAE reporting relationships in their respective organizations, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. The average ratings given by DAC members and CAEs are 4.4 and 4.6, respectively. Several DAC members and CAEs indicated that direct CAE reporting to deputy heads and direct CAE access to DACs (e.g., in camera meetings) ensure that CAEs and the internal audit function remain independent.

Two case studies were conducted to analyze the differences in CAE reporting relationships (i.e., CAEs who report directly to the Deputy Head in all aspects of their work, and those who do not). The case studies included a review of available documentation; investigation into the results of focus groups involving CAEs, deputy heads, DAC members and internal audit staff; and telephone interviews with selected CAEs and deputy heads who are involved in different types of reporting relationships. Investigations into situations where the CAE reports “solely and substantively” rather than “solely and exclusively” to the Deputy Head (as indicated in MAF ratings) indicated that, in most cases, the CAEs report administratively to managers other than the Deputy Head on minor aspects, such as approval of annual leave, that do not significantly impair the independence of the CAE. The key finding of the two case studies is that it is critical that the CAE report directly to the Deputy Head to ensure the independence of the internal audit function from line management. Another finding of the case studies is that it is not acceptable for the CAE to report to someone other than the Deputy Head because this lack of a direct reporting relationship impairs the ability of the CAE and the internal audit function to be independent, objective and free of influence from line management. In addition, the case studies indicated that it is important that CAEs have unfettered and direct access to their DAC in order to further ensure the independence of the internal audit function.

The findings of the focus groups involving CAEs, DACs and LDA deputy heads were similar to the survey and case study findings, particularly regarding the need for the CAE to report directly to the Deputy Head. Several CAE focus group participants also indicated that it is critical for the CAE to be at the management table so that he or she is attuned to the strategic focus of the department and internal audit staff are kept informed of the highest risk areas within the department. Although having a CAE at the management table may erode the independence of the internal audit function, the focus groups seemed to agree that the benefits related to having the CAE at the management table outweigh the potential risks related to the reduced independence of the internal audit function. The focus group findings are similar to the conclusions in the 2004 November Report of the Auditor General of Canada, chapter 1, “Internal Audit in Departments and Agencies,” which states that “internal audit contributes to better governance when it assumes a strategic orientation by working closely with the audit committee and senior management to address organization-wide risk, governance and control issues.”

Based on a comparative analysis, Canada has progressed further than the United Kingdom and Australia in ensuring the independence of the CAE by stipulating that CAEs report directly to deputy heads. As previously indicated, CAEs in the Government of Canada report directly to their Deputy Head in almost all cases. In Australia, the majority of, but not all, heads of internal audit have a formal reporting line to the Deputy Head, whereas in the United Kingdom, some reporting relationships fall short of direct. The HM Treasury report, Internal Audit Strategic Improvement Plan: Research Summary, published in January 2010, stated that 31 per cent of the 46 heads of internal audit surveyed in the United Kingdom indicated that their main reporting line was to the accounting officer, and 49 per cent indicated that there was a need for the accounting officer to be more actively engaged with internal audit.

The Policy states that deputy heads of LDAs are responsible for establishing an independent DAC that is comprised of a majority of external members who have been recruited from outside the federal public administration. According to the CAEs surveyed and based on a review of CAE reports, DAC reports and OCG documentation, all 46 of the large federal government organizations designated as LDAs under the Policy have established DACs that have some external members. There are two cases where two organizations share the same DAC comprised of external members (Transport Canada and Infrastructure Canada; and the Natural Sciences and Engineering Research Council and the Social Sciences and Humanities Research Council). In addition, the SDAAC was established to have external members in order to provide advice on horizontal and other audits of SDAs conducted by the OCG. As of October 2010, 152 external members have been appointed to these DACs by the Treasury Board (27 of which consist of external members who were appointed to two DACs).

According to the CAEs surveyed and the documentation reviewed, all DACs except one currently have a majority of external members. In most cases, the DAC has three external members, although the number of external DAC members ranges from one to four in the organizations surveyed. Each DAC typically has one or two internal members, one of which is the Deputy Head. Approximately 83 per cent of the CAE respondents stated that their Deputy Head attends the DAC meetings; 17 per cent indicated that their Deputy Head does not attend the DAC meetings. The CAEs and CFOs also typically attend the DAC meetings as ex officio participants.

DAC members, CAEs and LDA CFOs were asked to indicate to what extent there is independence from line management (exclusive of the Deputy Head) in DAC reporting relationships and the composition of DACs in their organization, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. As indicated in Figure 4, the average rating given by the three respondent groups is 4.6 or 4.7, indicating that DAC members are very independent from line management. The increase in DAC members’ independence is significant, given that the DACs that existed prior to the Policy consisted primarily of senior managers from within the same department.

Figure 4. Extent of Independence of DAC Members
Figure 4. Extent of Independence of DAC Members. Text version below:
Figure 4. Extent of Independence of DAC Members - Text version

The bar graph presents the perceptions of three interviewed groups on the independence of Departmental Audit Committee members. Results were as follows: Chief Audit Executives 4.7, Chief Financial Officers of large departments and agencies, 4.6, and Departmental Audit Committee Members, 4.6.

According to a report prepared by the OCG in October 2010, 94 per cent of external members appointed to DACs have been members of corporate boards of directors or other audit committees, as indicated in Table 1. This report also indicated that 42 per cent of external DAC members are former senior federal public servants.

Table 1. Experience of External DAC Members
Competencies Percentage of DAC Members
Corporate directorship or audit committee member 94
Governance 93
Department, industry or stakeholder specific 71
Public administration 62
Finance, accounting or auditing 56
Former senior federal public servant 42

The Policy states that the Deputy Head or an external member will chair the DAC and that, as the committees evolve, the preferred model is for the chair to be an external member. According to the CAEs surveyed and the documentation reviewed, 30 LDAs, or 65 per cent, have a DAC that is chaired by an external member; in most other cases, the chair of the DAC is the Deputy Head of the department or agency. According to the deputy heads surveyed, some of the DACs that are currently chaired by a deputy head will soon change to having an external chair.

Table 2. Chair of Departmental Audit Committees
Chair of DAC Number of LDAs Percentage of Total
External DAC member 30 65
Deputy head 15 33
Other internal member 1 2
Total 46 100

When deputy heads of LDAs and CAEs were asked to indicate whether they agree that the chair of their organization’s DAC should eventually be an external member, 67 per cent of the deputy heads and 78 per cent of the CAEs indicated that the chair should eventually be an external member (see Figure 5).

Figure 5. Who Should Chair the DAC?
Figure 5. Who Should Chair the DAC?. Text version below:
Figure 5. Who Should Chair the DAC? - Text version

The bar graph shows that 67% of Deputy Heads and 78% of Chief Audit Executives believed that the DAC chair should eventually be an external member. Additionally, 22% of the Deputy Heads of large departments and agencies and 12% of Chief Audit Executives from large departments and agencies believed that the chair of DACs should be the Deputy Head. Approximately 11% of Deputy Heads of large departments and agencies and 10% of Chief Audit Executives from large departments did not know who should chair DACs.

The following are the most frequent reasons given by deputy heads for which an external chair is the preferred model:

  • Provides a more independent perspective and diversity of opinion;
  • Eliminates the possibility that the deputy head could be accused of manufacturing his or her own advice; and
  • The optics of having an advisory committee chaired by a deputy head giving advice to a deputy head are not appropriate.

In contrast, the most frequent reasons given by deputy heads for which a deputy head should be the chair of the DAC are as follows:

  • It does not matter whether the Deputy Head is chair because the DAC is advisory.
  • The Deputy Head may not attend DAC meetings if he or she is not the chair, and it is highly valuable for the Deputy Head to interact with external members.
  • The Deputy Head has better knowledge of the organization.

Based on the documentation review, interviews with LDA deputy heads, and surveys of CAEs, the following three alternative DAC models are currently employed in the federal government:

  • The Deputy Head is chair of the DAC.
  • An external member is chair of the DAC.
  • The Deputy Head does not attend DAC meetings.

Each of these DAC models was the subject of a case study. The purpose of these studies was to analyze the different ways in which DACs operate in order to determine the effects of different types of DACs. The case studies included a review of relevant documentation; an examination of the results of focus groups involving DAC members, deputy heads and CAEs; and discussions with CAEs, DAC members and deputy heads who were involved with each DAC model. The key finding of the study was that the preferred model is for a DAC to have an external chair. Most of the CAEs, DAC members and deputy heads who were interviewed on the different DAC models stated that a DAC that has an external chair is preferable because it is more independent, ensures a diversity of opinion, and is viewed as more credible (i.e., the optics of having an advisory committee chaired by a deputy head giving advice to a deputy head are not appropriate).

Another key finding of the case studies is that it is important for the Policy to remain flexible and to allow for the evolution to a DAC that has an external chair in the organizations where that is not the case. Several of the CAEs, DAC members and deputy heads that were interviewed on the different DAC models stated that a transition period is necessary and that there are benefits to having a deputy head as the DAC chair. The main benefits stated were the value of deputy head interaction with external members and the level of deputy head engagement when chairing DAC meetings.

The findings of the focus groups involving CAEs, DACs and LDA deputy heads were similar to the case study findings. In particular, most focus group participants stated that the preferred model is for the chair of the DAC to be an external member, but they indicated that the Policy should remain flexible to allow for a gradual transition to an external chair.

As indicated in Table 3, all three jurisdictions (i.e., Canada, the United Kingdom and Australia) have established DACs that have external members in some of their departments. In the United Kingdom, the DAC contains only external members; in Canada, most DAC members are external (with the exception of one DAC). In Australia, less than one third of the DACs have a majority of external members. In the United Kingdom, the DAC chair is always an external member, whereas in Canada, 65 per cent of DAC chairs are external members, and about one half of DAC chairs in Australia are external members. Therefore, Canada is consistent with the other jurisdictions investigated that have DACs with external members for the purpose of enhancing the independence of their internal audit function. However, the United Kingdom has made the most progress in this regard because its DACs contain only external members.

Table 3. Comparison of DACs in Three Jurisdictions
Aspect of DAC Canada United Kingdom Australia
Which organizations are required to have a DAC with external members? Largest federal government departments and agencies, and agents of Parliament All departments, executive agencies and arm’s length bodies Departments and agencies under the Financial Management and Accountability Act should have one external member, preferably more
Does the DAC have internal members? Yes No Yes
What proportion of DACs have a majority of external members? All except one All Less than one third
What proportion of DAC chairs are external members? About 65 per cent All About one half
Who can be a DAC member?
  • Can include former deputy ministers and associate deputy ministers from within the department or elsewhere
  • One member must be a financial expert
  • Board member or independent external member
  • Can be a retired accounting officer but not from the department he or she served (infrequent occurrence)
  • One member should have financial experience
  • At least one member should have accounting or financial management experience
  • Broad business experience
  • Public sector experience
  • Understanding of the business or industry in which the entity operates
What is the role of the DAC? Advisory to Deputy Head Support to management board and Deputy Head Advisory to Deputy Head
Is there an external management board? No Yes—Minimum of three external members and chaired by the Minister’s secretary No

The Policy states that, although its principles also apply to seven agents of Parliament and the Public Service Commission of Canada, deputy heads of these organizations may authorize any departures from specific Policy requirements as they deem appropriate in light of the governance arrangements, statutory mandate and risk profile of the organization. The five agents of Parliament that participated in the evaluation, as well as the Public Service Commission of Canada, have appointed external DAC members and designated a CAE. According to the documentation reviewed, there are 19 external members appointed to the DACs of the seven agents of Parliament and the Public Service Commission of Canada. The five agents of Parliament that participated, as well as the Public Service Commission of Canada, have a DAC that has a majority of external members. The Deputy Head is the chair of the DAC in half of these organizations; the other half have an external DAC chair.

Conclusion: The Policy has contributed to increasing the independence of the internal audit function from line management across government by requiring DACs to have a majority of external members and a CAE position that reports directly to the deputy head of large federal government departments and agencies.

Evaluation Question 6: To what extent has the Policy contributed to improved risk management, governance, internal control and stewardship of resources across government? To what extent have the DACs contributed to improved risk management, governance and internal control across government? See footnote 2

Stakeholders were asked to indicate the extent to which the Policy, including the contribution of the DAC, has contributed to improved risk management, governance, internal control and stewardship of resources in the organization(s) with which they are involved, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. As shown in Table 4, all respondent groups indicated that the implementation of the Policy has contributed in some way to each of these four intended policy outcomes.

Table 4. Policy Contribution to Improved Risk Management, Governance, Internal Control and Stewardship of Resources
Stakeholder group Average Rating of Policy Contribution
Improved Risk Management Improved Governance Improved Internal Control Improved Stewardship of Resources
DAC members 4.0 3.8 3.7 3.5
LDA deputy heads 3.9 3.8 3.7 3.6
CAEs 3.8 3.7 3.7 3.3
Internal audit staff 3.9 3.6 3.3 3.1
LDA CFOs 3.7 3.6 3.7 3.1
Key informants 3.6 3.5 3.6 3.5
Departmental ADMs 3.4 3.2 3.4 3.1
SDA CFOs 2.8 2.8 3.3 2.5
SDA deputy heads 2.7 2.5 3.3 2.4
Average 3.5 3.4 3.5 3.1

As indicated in Table 4, respondents indicated that the Policy has contributed the most to improved risk management (3.5) and internal control (3.5); it has contributed the least to improved stewardship of resources (3.1). Several respondents, particularly DAC members, indicated that the least amount of attention has been paid to the stewardship of resources. Although the Policy has made some contribution to improving the stewardship of resources, the lower contribution may be explained by the relative state of maturity of the DACs and internal audit governance. Based on feedback obtained from respondents, the Policy’s major impacts on the improved stewardship of resources relate to DAC members’ reviews of Department Performance Reports (DPRs) and Reports on Plans and Priorities (RPPs), and the increased focus on the stewardship of resources in some of the audits performed.

In general, the respondent groups that provided the highest average ratings regarding the Policy’s contribution were LDA deputy heads, DAC members and CAEs. The ratings provided by ADMs are in the midpoint range between the ratings of other respondent groups. The two respondent groups that provided the lowest average ratings are SDA deputy heads and SDA CFOs, both of which indicated that the Policy somewhat contributed to each of the four intended Policy outcomes. The lower ratings given by SDA respondents can be attributed to the fact that SDAs do not have a DAC, a CAE or internal audit staff. Several SDA respondents also indicated that they have not been involved in many SDA horizontal audits conducted by the OCG. As previously indicated, the 50 SDAs are small organizations that collectively account for less than one per cent of total federal government expenditures. Consequently, it is appropriate for the Policy to place the highest priority on improving risk management, governance, internal control and stewardship of resources in LDAs, which account for more than 99 per cent of federal government expenditures.

Several respondent groups (LDA deputy heads, CAEs, and DAC members) were asked to distinguish between the contribution of the DAC and that of the other aspects of the Policy in terms of improved risk management, governance, internal control and stewardship of resources in their organization. Based on the average ratings of each respondent group, LDA deputy heads and CAEs indicated that the establishment of DACs that have external members is one of the most important aspects of the Policy; DAC members indicated that the contribution of DACs has been similar to that of the other aspects of the Policy with regard to achieving the intended Policy outcomes (see Figure 6).

Figure 6. DAC Contribution Compared With the Contribution of the Overall Policy to Improved Risk Management, Governance, Internal Control and Stewardship
Figure 6. DAC Contribution Compared With the Contribution of the Overall Policy to Improved Risk Management, Governance, Internal Control and Stewardship. Text version below:
Figure 6. DAC Contribution Compared With the Contribution of the Overall Policy to Improved Risk Management, Governance, Internal Control and Stewardship - Text version

The bar graph illustrates the contribution by the DAC and the rest of the Policy to improved risk management, governance, internal control and stewardship of resources to organizations from the perspective of three groups. Results from the three were as follows: Deputy Heads of large departments and agencies scored DACs’ contributions as 4.1 and the policy’s contributions as 3.7. DAC members scored the contributions of DACs as 3.8 and the contributions of the policy as 3.8. CAEs reported similar findings in their views of DACs but differed in the scores given to the policy, as shown by their 3.6.

The most frequent responses provided by respondents are outlined as follows.

Improved Risk Management
  • The establishment of a risk-based audit plan as required by the Policy has increased the focus of internal audits on areas of higher risk and importance.
  • Audits place more emphasis on risk management.
  • Several DACs have made risk management one of their priorities and have made significant contributions to enhancing the risk based audit plan and the corporate risk profile.
  • The independent and fresh perspective offered by external DAC members has helped senior managers of LDAs better assess what constitutes risk in their organization and how risk is defined, and focus on measures to mitigate and control identified risks.
  • Although some improvements have been made to the risk profile, several DAC members have indicated that it will be some time before full potential is realized.
  • The Policy’s contribution is complementary because it is only one of the factors leading departments to improved risk management.
Improved Governance
  • The largest contributions have been made in improving the governance of the internal audit function by establishing a more professional internal audit function that operates in accordance with the Policy, improving the oversight of the internal audit function with a qualified CAE and DAC members who have audit expertise, ensuring strong deputy head support for the internal audit function, and ensuring more timely implementation of internal audit recommendations, leading to improved management and business practices.
  • A number of respondents indicated that more attention is given to governance in audits.
  • DACs have significantly improved the process for following up on management responses to internal audit findings, and have contributed to ensuring that internal audit resources are directed to priority risk areas.
  • Most DACs have not focused extensively on governance outside the internal audit function, but some will devote more attention to this area in the future.
  • Because there are other initiatives driving improved governance, the Policy is complementary. However, the influence of internal audit on improved governance is more pronounced than it would have been had the Policy not been put in place.
Improved Internal Control
  • Because of their external, independent view backed by relevant, extensive knowledge and experience, DACs review the financial statements regularly and challenge departmental financial statements.
  • Audits of internal controls are performed regularly. Had the Policy not been developed, these audits would not have been carried out as frequently or as professionally.
  • DACs consider the department’s internal controls when examining each audit engagement, and management responses and action plans.
  • Implementation of the Policy has helped identify weaknesses and has strengthened internal controls in a number of areas.
  • The Policy’s emphasis on risk-based audit plans and assurance reporting has led to the examination of certain areas that were previously neglected to a certain extent.
  • The Policy is complementary to a number of initiatives in this area, including the move to auditable financial statements and the Policy on Internal Control.
Improved Stewardship of Resources
  • A number of DACs review the RPPs and DPRs, and some DACs have provided comments to improve the clarity and usefulness of these reports (e.g., more relevant performance measures). Several DAC members indicated that there is potential for improving the current RPPs and DPRs.
  • A number of respondents indicated that increased focus has been devoted to the stewardship of resources in some of the audits performed.
  • Implementation of the Policy plays a complementary role in improving the stewardship of resources. Although internal audits are important in confirming that the stewardship of resources is adequate, other activities play a more significant role in ensuring the appropriate stewardship of resources (e.g., reviews conducted by the Office of the Auditor General, MAF reviews carried out by central agencies, Treasury Board approval of resources).

According to a documentation review, a number of DACs that have external members have been formed in the last two years. Therefore, it may be premature to assess the full potential of these DACs. As previously indicated, some DAC members stated that they have identified additional areas to focus on in the future and that it will be some time before the full potential of the DACs is realized.

As previously indicated, several respondents stated that the Policy is complementary to a number of other policies and initiatives (e.g., Policy on Internal Control, Risk Management Policy, Federal Accountability Act and the Policy Suite Renewal) that focus directly on aspects such as internal control and risk management. Because the intended outcomes of the Policy on Internal Audit are shared with these complementary programs, it is difficult to attribute the impact of the Policy. The responses provided in this section include only the impacts indicated by respondents as attributable to the Policy.

Conclusion: The implementation of the Policy, particularly the establishment of DACs that have external members, has improved risk management, governance, internal control and the stewardship of resources in federal government departments and agencies.

Evaluation Question 7: To what extent has deputy heads’ confidence increased in the assurance provided by the internal audit function and the advice provided by DACs on risk management, control and governance? What are the nature and extent of other DAC contributions to strong and accountable public sector management? See footnote 3

LDA deputy heads were asked to rate the extent to which their confidence has increased in the advice provided by their DAC on risk management, control and governance processes in their organization, compared with the advice provided by the DAC that existed prior to the Policy. LDA deputy heads that did not hold the same position prior to the Policy were asked to rate their level of confidence in the advice provided by their DAC. The average rating given by LDA deputy heads was 4.3 out of 5. Two thirds of the LDA deputy heads stated that their level of confidence (or increased confidence) in the advice obtained from DACs was “significant” (or “to a great extent”). Nine, or 26 per cent, of the deputy heads did not provide a comment because, for example, they have not been in the position long enough or their DAC was only recently formed (see Figure 7).

Figure 7. LDA Deputy Heads’ Confidence in Advice Provided by DACs
Figure 7. LDA Deputy Heads’ Confidence in Advice Provided by DACs. Text version below:
Figure 7. LDA Deputy Heads’ Confidence in Advice Provided by DACs - Text version

The pie chart presents the findings relating to the extent that the Deputy Heads of large departments and agencies have increased confidence in the advice provided by their DAC regarding risk management, control, and governance processes in their organization as compared to the DAC that existed prior to the Policy. 41% of Deputy Heads felt a significant level of confidence, 26% a great extent, 6% felt somewhat confident, and 26% did not know or had no comment.

The following are some of the most frequent reasons given by LDA deputy heads for their high degree of confidence in the advice they receive from DACs:

  • External DAC members bring a different perspective to risk management, control and governance discussions, and foster a more comprehensive exploration of underlying issues and implications.
  • The level of confidence in the advice given to deputy heads by DACs has increased significantly since the establishment of DACs composed of external members because those that have a majority of external members provide a level of independence and objectivity that is invaluable.
  • The variety of DAC members’ experience from outside the public sector enables DACs to provide deputy heads with a perspective different from that obtained previously from a DAC composed of senior managers. The combined expertise and experience of external DAC members adds depth to the advice given to the Deputy Head, which was not feasible when the DAC consisted of a subset of the management team.
  • External DAC members have extensive experience and knowledge in the areas of risk management, control and governance.
  • The establishment of DACs that have external members is the single best improvement in public sector management in several years.

Several LDA deputy heads stated that, in addition to providing advice on risk management, control and governance, DACs have made other contributions to strong and accountable public sector management. Because of the extensive experience held by DAC members, many deputy heads stated that they use their DAC as a sounding board and a strategic resource for strengthening the overall institution, rather than just the internal audit function. This broader contribution made by DACs to strong and accountable public sector management constitutes a positive unintended outcome of the Policy.

Several deputy heads stated that their confidence in the assurance provided by the internal audit function has increased, and they attributed their increased confidence primarily to the establishment of DACs that have external members. Many deputy heads indicated that the oversight provided by these DACs has increased the professionalism, credibility and usefulness of the internal audit function. Some deputy heads indicated that the policy requirement stipulating that internal audit focus on delivering assurance rather than consulting services has resulted in a more independent and professional assessment function.

Conclusion: LDA deputy heads have a great deal of confidence in the assurance provided by the internal audit function and the advice provided by their DAC regarding risk management, control and governance processes.

Evaluation Question 8: To what extent has the Policy increased the effectiveness of the internal audit function across government? To what extent have DACs contributed to increasing the effectiveness of the internal audit function across government?

As shown in Table 5, the proportion of LDAs that obtained an “acceptable” or “strong” MAF rating for their internal audit function increased from 42 per cent in 2005–06 to 85 per cent in 2009–10.

Table 5. MAF Ratings of the Internal Audit Function in LDAs
Year MAF Rating Number of LDAs Rated
Attention Required Opportunities for Improvement Acceptable Strong
2005–06 19% 39% 39% 3% 38%
2006–07 23% 40% 32% 5% 40%
2007–08 10% 28% 57% 5% 39%
2008–09 2% 20% 71% 7% 41%
2009–10 2% 13% 68% 17% 46%

As shown in Table 6, in 2009–10, 89 per cent of LDAs obtained an “acceptable” or “strong” MAF rating for their internal audit governance structure; 76 per cent of LDAs received an “acceptable” or “strong” rating for their internal audit performance in compliance with the Policy; and 81 per cent of LDAs obtained an “acceptable” or “strong” rating for their progress in the use of audit results and the development of their audit capacity.

Table 6. MAF Ratings of the Internal Audit Function in LDAs in 2009–10
Criteria 2009–10 MAF Rating
Attention Required Opportunities for Improvement Acceptable Strong
Internal audit governance structure is in place 2% 9% 61% 28%
Internal audit work is performed in accordance with the Policy on Internal Audit and Directive on Internal Audit 2% 22% 56% 20%
Progress is being made in the use of audit results and the continued development of the audit capacity 4% 15% 61% 20%

Stakeholders were asked to indicate the extent to which the Policy has increased the effectiveness of the internal audit function, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. As shown in Figure 8, the highest average ratings were provided by CAEs (4.3), DAC members (4.1) and LDA deputy heads (4.0). The two respondent groups that gave the lowest average ratings were SDA deputy heads (3.5) and SDA CFOs (3.3). The lower rating can be attributed partially to the fact that SDAs do not have a DAC, CAE or dedicated internal audit staff, and several SDAs indicated that they have participated in only a limited number of horizontal audits conducted by the OCG.

Figure 8. Increased Effectiveness of the Internal Audit Function
Figure 8. Increased Effectiveness of the Internal Audit Function. Text version below:
Figure 8. Increased Effectiveness of the Internal Audit Function - Text version

The bar graph maps key informants’ perceptions on the extent to which the Policy has increased the effectiveness of the internal audit function. Scores were as follows: Chief Audit Executives, 4.3, Departmental Audit Committee Members, 4.1, Deputy Heads of large departments and agencies, 4.0, Chief Financial Officers of large departments and agencies, 3.8, Internal Audit staff, 3.7, Assistant Deputy Ministers, 3.5, Deputy Heads of Small departments and agencies, 3.5, and Chief Financial Officers of small departments and agencies, 3.3.

The respondents who indicated an increase in the effectiveness of their internal audit function were also asked to indicate to what extent DACs with external members have contributed to increasing the effectiveness of the internal audit function in their organization. As indicated in Figure 9, the highest rating, at 4.5 out of 5, was provided by LDA deputy heads.

Figure 9. DAC Contribution to the Increased Effectiveness of the Internal Audit Function
Figure 9. DAC Contribution to the Increased Effectiveness of the Internal Audit Function. Text version below:
Figure 9. DAC Contribution to the Increased Effectiveness of the Internal Audit Function - Text version

The bar graph illustrates the extent that DACs with external members have contributed to the increased effectiveness of the internal audit function within organizations. The highest scores were provided by the Deputy Heads of large departments and agencies with 4.5, then by Chief Audit Executives with 4.3. A score of 4.1 was provided by both the Assistant Deputy Ministers and Chief Financial Officers of large departments and agencies. The lowest score of 3.7 was provided by internal audit staff.

The following are the most frequent reasons given by respondents for the increased effectiveness of the internal audit function:

  • The quality of internal audit and departmental managers’ attention to internal audit recommendations have been strengthened by the clear delineation of the CAE’s role under the Policy and the requirement stipulating that the DAC conduct an external review of internal audit reports and management responses.
  • The policy requirement for a risk-based internal audit plan ensures that internal audits focus on areas of highest importance or risk to the organization.
  • The policy requirement stipulating that the internal audit function focus on delivering assurance rather than advisory services has increased the effectiveness of the internal audit function.
  • The Policy has helped improve the rigour and professionalism of the internal audit function.
  • The development and adoption of Government of Canada audit standards as a result of the Policy’s implementation provide a solid framework for internal audit practices across government.
  • The establishment of DACs that have external members has contributed to increasing the rigour in internal audit, as well as in management responses, implementation and monitoring, by providing useful outside views and a challenge function.
  • External DAC members challenge the internal auditors to focus on areas of high risk and to take into account the work of external auditors when planning which areas to audit.
  • Additional funding to staff the internal audit function in departments has increased the number of internal auditors and resulted in more comprehensive audit coverage.

Several SDA respondents stated that the horizontal audits of SDAs conducted by the OCG have contributed to increasing the effectiveness of internal audit as it relates to SDAs. Since most SDAs do not have an internal audit function, several respondents indicated that horizontal audits conducted by the OCG are an effective tool for meeting their audit needs. Several SDA respondents also indicated that, given the resource constraints in SDAs, the horizontal audits have ensured an internal audit capacity in key areas of management.

Respondents were asked to indicate the appropriateness of the selection and appointment processes (e.g., level of departmental and OCG involvement), skill sets and previous employment (e.g., private and public sector experience), mandate, terms of service, compensation and costs of DAC members in their organization, on a scale of 1 to 5, where 1 is not at all appropriate, 3 is somewhat appropriate, and 5 is very appropriate. As indicated in Table 7, the highest average rating given by all respondents was for the criterion regarding the skills sets and previous employment of DAC members.

Table 7. Appropriateness of DAC Selection Process, Qualifications and Mandate
Criteria Type of Respondent
LDA Deputy Heads CAEs CFOs DAC Members Key Informants Average
Skills sets and previous employment 4.5 4.4 4.3 4.5 4.3 4.4
Mandate 4.4 4.2 4.0 4.4 3.9 4.2
Terms of service 4.4 4.3 3.9 4.3 3.9 4.2
Selection and appointment process 3.7 4.1 4.1 4.3 4.2 4.1
Compensation and costs 4.2 4.1 3.8 3.9 3.8 4.0

Almost all respondents stated that OCG and departmental staff involved in the joint selection process performed an excellent job in selecting the most qualified candidates for DACs, and that the high calibre of DAC members has contributed to the overall success of the Policy. Most respondents stated that their DAC members have a combination of private and public sector skills and that this balance is necessary for an effective DAC. Several respondents indicated that, ideally, a DAC that has three external members should have one member who fulfils each of the following qualifications:

  • Private sector experience, preferably in a field related to the operations of the department or agency;
  • Financial or auditing experience and qualifications, preferably as a member of a corporate audit committee or corporate board of directors; and
  • Previous experience as a deputy minister of a federal government department.

Most respondents stated that it is critical for one of the external DAC members to be a former federal deputy minister because it is important for the DACs to understand the differences between the public and private sector. However, several respondents indicated that there should be a limit to the number of former federal government employees who are DAC members (i.e., one per DAC) because DACs need fresh and external perspectives.

Based on a comparative analysis with other jurisdictions regarding the qualifications of external DAC members, Canada is similar to Australia and the United Kingdom in requiring that one member possess financial management experience. In the United Kingdom, external board members can be former civil servants, but this does not happen frequently. It is permitted in Australia. As previously indicated, 42 per cent of external DAC members in Canada are former senior federal public servants. Canada is similar to Australia in this respect, but has not gone to the same extent as the United Kingdom in restricting the presence of former civil servants on DACs. Several respondents in the focus groups involving DAC members and LDA deputy heads stated that, although it is useful to have one DAC member who has federal government experience, the number of former senior federal public servants serving as DAC members should be limited to ensure that the DACs are able to provide a perspective based on experience outside the federal government.

As indicated in Table 7, LDA deputy heads gave the lowest average rating for the DAC selection and appointment process (3.7). According to several LDA deputy heads, the main reason for this is that that the DAC candidate approval process is very time consuming and unpredictable. For example, one respondent indicated that it took eight months to get one DAC candidate approved. Another respondent indicated that his or her department was forced to deal with vacancies in its DAC for too long. Several respondents stated that the current approval process needs to be streamlined. A number of respondents indicated that some prospective candidates suggested by the OCG and the department have not received ministerial approval. A number of CAEs and deputy heads stated that some existing DAC members are dissuaded from renewing their DAC membership and some potential DAC candidates have decided not to pursue DAC membership as a result of the unpredictability of the ministerial approval process.

Several participants in the focus groups involving DAC members, LDA deputy heads and CAEs also expressed concerns about the unpredictability of the process and the amount of time required to obtain approval of external DAC members. The current delays are out of the OCG’s control and result from the considerable time required to obtain Treasury Board ministerial approval. Some focus group participants suggested that it was not necessary to obtain Treasury Board ministerial approval and that this process should be delegated to the OCG or the Secretary of the Treasury Board for streamlining purposes.

As indicated in Table 7, all respondent groups indicated that they are satisfied with the DAC’s mandate. Several respondents stated that the mandate specified in the Policy and the activities actually carried out by DACs are appropriate and provide sufficient flexibility for the DACs to focus on the specific priorities of the department or agency with which they are involved. A number of respondents, specifically LDA deputy heads, indicated that the DAC’s mandate should not be expanded. A small number of respondents stated that the DAC mandate should not include the review and approval of program evaluation reports because this would take the DAC’s attention away from its existing mandate.

The mandate of DACs in Canada was compared with the mandates of DACs in other jurisdictions. The comparison revealed that Canada is consistent with other countries because the mandate in the other jurisdictions studied is limited primarily to providing assurance. In Canada and Australia, the DAC’s role is to provide advice to the Deputy Head. However, in the United Kingdom, the DAC’s role is to support the management board and the Deputy Head. As stated in the document Corporate Governance in Central Government Departments: Code of Good Practice, each department in the United Kingdom should be managed by an effective management board. A recent protocol document states that these management boards should be chaired by secretaries of state and that the boards should include the following: two to four ministers; three or four senior officials, including the permanent secretary and the finance director; and three or four non-executive board members, the majority of whom should be drawn from the commercial private sector and should have, among them, experience in managing large organizations. The HM Treasury Audit Committee Handbook states that the role of the audit committee is to support the board and the accounting officer by reviewing the comprehensiveness of assurances in meeting the needs of the board and the accounting officer in this area, and by reviewing the reliability and integrity of the assurances provided. Prior to May 2010, the Deputy Head (permanent secretary) was responsible for chairing the management board of each central government department.

LDA deputy heads, CAEs and DAC members were asked to indicate the appropriateness of the Policy regarding the CAE selection and appointment processes (e.g., level of departmental and OCG involvement), skill sets and previous employment, and mandate, on a scale of 1 to 5, where 1 is not at all appropriate, 3 is somewhat appropriate, and 5 is very appropriate. Based on the average ratings, all three respondents groups indicated that the mandate, the selection and appointment processes, and the skill sets and qualifications as set out in the Policy, are appropriate.

Table 8. Appropriateness of CAE Selection Process, Qualifications and Mandate
Criteria Type of Respondent
LDA Deputy Heads CAEs DAC Members Average
Mandate 4.2 4.3 4.4 4.3
Selection and appointment processes 4.5 4.1 4.1 4.2
Skill sets and qualifications 4.3 4.0 4.3 4.2

The following are the most frequent responses received regarding the CAE selection and appointment processes, skill sets and qualifications, and mandate:

  • Most LDA deputy heads and DAC members indicated that they are satisfied with the qualifications and capabilities of the CAE in their organization.
  • Several respondents felt that the OCG should continue to be involved in the CAE selection and appointment processes but that the final decision should rest with the department in order to ensure a good organizational fit.
  • Several respondents stated that all CAEs should be required to obtain a CIA designation; a number of other respondents indicated that a CIA designation is not required if an individual already has an accounting designation (e.g., Chartered Accountant, Certified Management Accountant).
  • Because there is a shortage of qualified CAEs in the federal government, some respondents indicated that flexibility is required in the CAE selection and appointment processes in order to reflect this situation.
  • Some respondents stated that, in addition to audit experience, CAEs require management skills and an in-depth understanding of the department in order to become senior management committee members.

The November 2004 Report of the Auditor General of Canada, Chapter 1, “Internal Audit in Departments and Agencies,” provides similar insight into the last observation regarding the qualifications required by CAEs. The report states the following:

Internal audit contributes to better governance when it assumes a strategic orientation by working closely with the audit committee and senior management to address organization wide risk, governance, and control issues. To be effective, internal audit groups need to move from a tactical level to a strategic level. They need to align their resources and provide assurance on risk, governance, and control of business processes that support the organization’s objectives and that demonstrate the value that internal audit adds.

Several participants in the focus groups involving DAC members, LDA deputy heads and CAEs stated that it is critical for the OCG to be involved in CAE selection. Other focus group participants indicated that clarification is needed on the qualifications that are required by CAEs, including the steps to be taken to enforce the stipulated requirements.

Based on the documentation review, the 2001 Policy on Internal Audit stated that internal audit reports should be made accessible to the public, and the 2006 Policy on Internal Audit specified that completed internal audit reports should be posted on departmental websites in a timely manner. A review of departmental websites indicates that internal audit reports are being posted regularly. Pretest interviews with a limited number of key informants and stakeholders revealed that there are advantages and disadvantages to making audit reports public. The main advantage mentioned by key informants and stakeholders was that making audit reports public increases the accountability and transparency of government. In addition, the increased visibility of internal audit reports ensures that follow-up action is taken because the public and the media are aware of the actions and can request information on their status. The key disadvantages mentioned by some key informants and stakeholders were inappropriate media attention, excessive effort devoted to writing internal audit reports, and a disincentive to perform audits in sensitive areas. Overall, however, most respondents agreed that the advantages of making audit reports public outweigh the disadvantages and that making audit reports public contributes to the effectiveness of the internal audit function.

As stated in Internal Audit in the Government of Canada: Jurisdictional Summary, changes were made to the Access to Information Act as part of the Federal Accountability Act that was enacted in December 2006 to protect internal audit working papers for a period of up to 15 years. These changes were made to safeguard the candour of the internal audit process and to avoid the release of incomplete information that could be misconstrued by the press and the public. Several key informants also indicated that extending the period for which internal audit working papers are protected has been useful in increasing the effectiveness of internal audit in the federal government.

Conclusion: Implementation of the Policy has significantly increased the effectiveness of the internal audit function across government.

Evaluation Question 9: To what extent has the Policy resulted in increased management action on internal audit recommendations, leading to improved risk management, governance and control in audited areas?

Respondents were asked to indicate the extent to which managers have implemented internal audit report recommendations in their respective organizations as a result of the Policy, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. As shown in Figure 10, all respondents groups indicated that there has been some increase in management action on internal audit report recommendations as a result of the Policy. The highest average ratings were provided by DAC members (4.2), CAEs (4.0), LDA deputy heads (3.9) and key informants (3.9). The lowest average ratings were provided by SDA deputy heads (3.1) and SDA CFOs (3.3). A contributing factor to the lower ratings provided by SDA representatives was the lack of SDA involvement in horizontal audits conducted by the OCG.

Figure 10. Management Implementation of Internal Audit Report Recommendations
Figure 10. Management Implementation of Internal Audit Report Recommendations. Text version below:
Figure 10. Management Implementation of Internal Audit Report Recommendations - Text version

The bar graph provides views on the extent of management implementation of internal audit report recommendations as a result of the policy. In descending order, DAC members noted a 4.2, Chief Audit Executives a 4.0, Deputy Heads of large departments and agencies a 3.9, key informants a 3.9, Assistant Deputy Ministers a 3.5, Chief Financial Officers of large departments and agencies a 3.5, Internal Audit staff a 3.4, Chief Financial Officers of small departments and agencies a 3.3, and Deputy Heads of small departments and agencies a 3.1.

The most frequent reasons given for the increase in management action on internal audit report recommendations as a result of the Policy are as follows:

  • CAEs and internal audit staff have been tracking the implementation of internal audit recommendations more formally since the Policy was introduced.
  • DACs have taken a strong interest in managers’ implementation of action plans and have emphasized this in dealing with managers.
  • Because DAC members challenge managers on the feasibility of their action plans, managers takes their action plans more seriously.
  • DACs regularly review follow-up actions on audit recommendations and challenge managers on outstanding items.
  • Given the DAC’s high profile and its reporting relationship to the deputy head, managers take implementation of internal audit report recommendations more seriously.
  • There has been a significant increase in the number of recommended actions that have been implemented since the Policy came into effect.

The average ratings provided by ADMs (3.5), LDA CFOs (3.5) and internal audit staff (3.4) are in the middle of the average ratings provided by the other respondents groups. Although these groups indicated on average that the Policy has resulted in a considerable increase in management action on internal audit report recommendations, the most frequent reason given by these respondents as to why the Policy has not had a greater impact is that management implementation in their organization was already high prior to the Policy. A number of respondents also indicated that, because they have just implemented systematic follow-up procedures, it is too soon to tell whether management action on internal audit report recommendations has increased.

A review of CAE and DAC reports indicates that most LDAs regularly follow up on and monitor management action on internal audit report recommendations. Figure 11 provides an example of the follow up actions taken by one department (Foreign Affairs and International Trade Canada) and indicates that, for this department, the completion rate on management actions to address internal audit recommendations has increased from 68 per cent as of December 2008 to 92 per cent as of January 2010. This example is relevant because it is one of the few figures included in CAE and DAC reports that provides current and baseline quantitative data to illustrate the trend in the management action completion rate.

Figure 11. Management Action Completion Rate in Foreign Affairs and International Trade Canada
Figure 11. Management Action Completion Rate in Foreign Affairs and International Trade Canada. Text version below:
Figure 11. Management Action Completion Rate in Foreign Affairs and International Trade Canada - Text version

The trend graph for the department of Foreign Affairs and International Trade demonstrates that in December 2008, 68% of recommendations had been addressed, 10% were in progress, and 10% were delayed. In April 2009 approximately 85% were addressed, 1% were in progress, and 5% delayed. In June 2009 approximately 80% had been addressed, 1% were in progress, and 9% delayed. In September 2009, approximately 75% were addressed, 5% in progress, 1% delayed, and 10% not due. And in January 2010, approximately 92% were addressed, 1% delayed and 5% not due.

Respondents who indicated that management action on internal audit recommendations has increased were asked to also indicate the significance of the resulting increase in benefits, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is very significant. As indicated in Figure 12, the highest average ratings were provided by CAEs (3.9) and DAC members (3.9); the lowest average ratings were provided by SDA respondents (3.1) and LDA CFOs (3.1).

Figure 12. Increased Benefits Resulting From Greater Implementation of Management Action Plans
Figure 12. Increased Benefits Resulting From Greater Implementation of Management Action Plans. Text version below:
Figure 12. Increased Benefits Resulting From Greater Implementation of Management Action Plans - Text version

The bar graph shows the extent of increased benefits obtained from greater implementation of management action plans as a result of the Policy by stakeholder group.

In descending order Chief Audit Executives and DAC members scored 3.9, Deputy Heads of large departments and agencies, key informants, and Internal Audit staff a 3.5, Assistant Deputy Ministers a 3.3, and the Deputy Heads of small departments and agencies and all.Chief Financial Officers, 3.1.

The most frequent benefits mentioned by respondents are averted risks, improved operational efficiency, improved risk management, and cost savings. Some other benefits mentioned by respondents include stronger internal controls, reduced risks, and avoided costs. Some respondents indicated that it is difficult to measure the benefits that have been achieved; others stated that it is premature to assess the benefits achieved.

Conclusion: Implementation of the Policy has resulted in an increase in management action on internal audit report recommendations due primarily to a concerted focus by DACs and CAEs on following up on the implementation of management action plans.

Evaluation Question 10: To what extent has the Policy contributed to increasing the capacity of the internal audit function across government?

In keeping with one of the cornerstone objectives of the Policy on Internal Audit, the Treasury Board approved the progressive allocation of incremental resources to help the internal audit community implement the Policy’s new requirements. As indicated in Table 9, once the strategy related to reconfiguring the internal audit community is fully implemented, the total incremental funding across government is expected to rise to about $40.4 million on an ongoing basis. However, two budgeted expenditures indicated in Table 9 that have not been incurred are as follows: $9.7 million per year for an anticipated increase in the salaries of department internal audit staff resulting from a reconfiguration of the internal audit community in the federal government; and $1.3 million per year for technological support to select and implement a common audit information management platform to standardize internal audit work processes. Although some progress has been made on both of these activities, payment of these budgeted expenditures has not yet been required. By removing these two expenditures from the total of $40.4 million that was initially approved, the amount approved on an ongoing basis for other departmental and OCG audit activities is approximately $29.4 million, of which $28.5 million was approved for departmental internal audit activities, and the remainder for OCG activities.

Table 9. Incremental Funding Accompanying the 2006 Policy on Internal Audit ($ millions)
2006-07 2007-08 2008-09
Departmental
Compensation for external DAC members 1.5 4.6 6.0
Additional internal audit staff salaries 4.7 12.1 17.8
Training, certification and professional membership 2.3 3.2 3.9
Agents of Parliament 0.0 1.0 0.8
Readiness assessments 2.7 0.0 0.0
Reconfiguration of internal audit community 0.0 0.0 9.7
Subtotal 11.2 20.9 38.2
Centrally Managed
Development of training programs for DACs 0.2 0.5 0.4
Development of training for internal audit practitioners 0.0 1.0 0.5
Technological support 1.5 1.4 1.3
Subtotal 1.7 2.9 2.2
Total 12.9 23.8 40.4

As shown in Figure 13, a survey conducted by Statistics Canada on behalf of OCG Capacity Building and Community Development revealed that the number of internal audit staff in the federal government (core public service indeterminate employees, excluding separate employers) has more than doubled, from 190 in 2005 to 479 in 2010.

Figure 13. Number of Internal Audit Employees in the Core Public Service
Figure 13. Number of Internal Audit Employees in the Core Public Service. Text version below:
Figure 13. Number of Internal Audit Employees in the Core Public Service - Text version

The bar graph maps the steady increase in the number of auditors from 2005 to 2010. Results for each year are as follows: in 2005, there were 100 auditors; in 2006, 225; in 2007, 280; in 2008 402; in 2009, 448, and in 2010 there were 479 auditors.

According to the November 2004 Report of the Auditor General of Canada, Chapter 1, “Internal Audit in Departments and Agencies,” the total budgeted expenditures for internal audit in the federal government were $54 million in 2002–03. All CAEs surveyed were asked to provide information on the actual internal audit expenditures and FTEs in their organization for fiscal years 2005–06 and 2009–10. The specific information requested was all FTEs involved in implementing the Policy, including all staff who spend more than 60 per cent of their time on internal audit, those involved with DACs and the CAE, and secretariat and management support staff. Financial information was provided by CAEs for 41 of the 46 LDAs. As indicated in Figure 14, the actual internal audit expenditures of the organizations surveyed increased by 75 per cent, from $47.2 million in 2005–06 to $82.6 million in 2009–10. The number of staff devoted to the internal audit function, including the administrative staff who support the DACs and CAEs, increased by 68 per cent, from 359 FTEs in 2005–06 to 602 FTEs in 2009–10, in the 41 organizations that provided information.

Figure 14. Change in Departmental Internal Audit Expenditures and FTEs Since the Policy Was Introduced
Figure 14. Change in Departmental Internal Audit Expenditures and FTEs Since the Policy Was Introduced. Text version below:
Figure 14. Change in Departmental Internal Audit Expenditures and FTEs Since the Policy Was Introduced - Text version

The two bar graphs offer a side by side comparison of Internal Audit expenditures with the number of staff dedicated to the audit function. Overall expenditures for 2005-6 were $47.2 million and in 2009-10 they were $82.6 million. The number of internal audit staff rose from 359 in 2005-6 to 602 in 2009-10.

CAEs, DAC members and internal audit staff were asked to indicate the extent to which the Policy has contributed to increasing the capacity of the internal audit function within their organization (e.g., increase in number and scope of internal audits), on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. The average ratings provided by CAEs, DAC members and internal audit staff were 3.7, 3.4 and 3.0, respectively. Although the majority of respondents indicated that the capacity of their internal audit function has increased to a certain extent, a large number indicated that the following factors have limited the increase in internal audit capacity:

  • There is a shortage of qualified and experienced internal auditors in federal government departments, which has limited the departments’ ability to increase internal audit capacity because the number, scope and quality of audit products are directly affected.
  • Although increased funding has resulted in more internal auditors, most of the new auditors are inexperienced and require supervision and ongoing training. Therefore, output has not increased proportionally to the increased investment.
  • Several respondents indicated that they have not increased the number of audits since the Policy was introduced, in part because of increased “overhead” requirements (e.g., preparation of risk-based audit plan, MAF reporting requirements). However, with the bulk of the foundational work now completed, they anticipate being able to produce more audits in the future.
  • Some CAEs and internal audit staff indicated that, although the quantity of audits has not risen, the quality of them has increased considerably. Some factors that respondents gave for the increase in quality include greater rigour, adherence to internal audit standards and a more risk-based approach.
  • Some CAEs and internal audit staff stated that the scope of audits have changed since the Policy was implemented.

Several participants in the focus group involving internal audit staff stated that providing support to their DAC, as well as meeting internal audit planning and reporting requirements (e.g., risk-based audit plans, CAE reports, DAC reports) and MAF reporting requirements, have resulted in fewer resources devoted to performing internal audits, particularly in organizations that have a limited number of internal audit staff. Some participants indicated that there are between one and four FTEs devoted to these overhead requirements, depending on the size of the organization. Most participants in the focus groups involving DACs, CAEs and LDA deputy heads acknowledged these overhead requirements, particularly the MAF reporting burden. Participants also indicated that foundational work, such as the preparation of risk-based audit plans, is necessary to provide direction for the internal audit function.

The number of internal audit reports received by the OCG does not appear to have changed significantly in recent years, as indicated in Table 10.

Table 10. Internal Audit Reports Received by the OCG
Year Number of Reports
2007-08 263
2008-09 256
2009-10 244

One factor that has limited the capacity of the internal audit function is the shortage of internal auditors. As shown in Figure 15, a survey conducted by the OCG in March 2009 indicated that 28 per cent of the funded internal audit positions in the core public service (37 organizations) were vacant.

Figure 15. Occupancy Status of Audit Positions in 2009
Figure 15. Occupancy Status of Audit Positions in 2009 . Text version below:
Figure 15. Occupancy Status of Audit Positions in 2009 - Text version

The following bar graph presents the results of a survey conducted by the Office of the Comptroller General in March 2009. It was found that 66% of internal audit positions were occupied, 28% vacant, 3% were occupied temporarily, and 2% were temporarily unoccupied.

According to the CAEs surveyed, several LDAs have increased their use of contracted resources to meet their audit requirements as a means of dealing with the shortage of internal auditors in the federal government. Based on the financial information provided by CAEs, total expenditures on professional internal audit services increased by 54 per cent, from $11.8 million in 2005–06 to $18.2 million in 2009–10, in the 37 LDAs that provided pertinent financial information.

Several key informants and CAEs indicated that horizontal audits of LDAs and SDAs conducted by the OCG have contributed to increasing the capacity of the internal audit function across government, particularly as it relates to SDAs. In 2009–10, the OCG spent approximately $1.8 million on horizontal audits. Given the resource constraints in SDAs, several SDA respondents indicated that the horizontal audits of SDAs conducted by the OCG have ensured an internal audit capacity in key areas of management in these organizations.

Conclusion: The Policy has contributed to an increase in the capacity of the internal audit function across government through the approval of $40 million annually in accompanying funding for departmental and OCG internal audit activities.

Evaluation Question 11: To what extent has the Policy contributed to audit coverage appropriate to the level of risk across government?

Respondent groups were asked to indicate to what extent the Policy has contributed to audit coverage appropriate to the level of risk in their organization, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. As indicated by the average ratings in Figure 16, the respondent groups that gave the highest ratings were CAEs (4.1), LDA deputy heads (4.0), DAC members (3.9) and key informants (3.9); the respondent groups that gave the lowest ratings were SDA deputy heads (3.1) and SDA CFOs (3.1).

Figure 16. Policy Contribution to Audit Coverage Appropriate to Level of Risk
Figure 16. Policy Contribution to Audit Coverage Appropriate to Level of Risk . Text version below:
Figure 16. Policy Contribution to Audit Coverage Appropriate to Level of Risk - Text version

The bar graph demonstrates the extent the Policy contributed to audit coverage appropriate to the level of risk for organizations. Results by stakeholder group are in descending order.

Chief Audit Executives a 4.1, Deputy Heads of large departments and agencies a 4.0, DAC members and key informants a 3.9, Chief Financial Officers of large departments and agencies a 3.6, Internal Audit staff a 3.5, Assistant Deputy Ministers a 3.4, Chief Financial Officers of small departments and agencies a 3.1, and Deputy Heads of small departments and agencies a 3.1.

The following are the most frequent comments received regarding the extent to which the Policy has contributed to audit coverage appropriate to the level of risk:

  • The risk-based audit plan is an excellent tool for providing assurance that the highest organizational risks have been investigated.
  • The Policy has facilitated adjustments to audit coverage in accordance with the corporate risk profile of the organization, with an increased capacity to audit the areas of highest risk.
  • The funding provided under the Policy has increased the capacity to perform more audits, thereby increasing audit coverage.
  • The discussion of risk has become more effective.
  • More clarity is needed regarding overall assurance before the appropriateness of audit coverage can be assessed.

As indicated in Figure 16, the average ratings provided by LDA CFOs, internal audit staff and ADMs fell in the median of the ratings provided by the other respondents. The following are the most frequent responses given by these respondents for their ratings:

  • Only the higher risk areas are audited.
  • The shortage of qualified auditors has resulted in a lack of sufficient audit coverage.
  • Coverage is increasing to the expected level.
  • Some organizations already had appropriate coverage prior to the Policy.

As indicated in Figure 16, the lowest average ratings (3.1) were provided by SDA deputy heads and SDA CFOs. The lower ratings can be explained by the following:

  • Several SDA representatives stated that horizontal audits of SDAs conducted by the OCG have improved their audit coverage because they did not previously conduct any internal audits.
  • Some SDA representatives indicated that audit coverage has been restricted by the limited number of horizontal audits in which they have participated.

As previously indicated, the Policy has contributed to increasing the capacity of the internal audit function across government through the approval of accompanying funding for departmental and OCG internal audit activities. This increased capacity has resulted in an increase in coverage.

A review of CAE reports, DAC reports and departmental websites indicated that almost all LDAs have prepared risk-based audit plans since the Policy was introduced. In addition, the OCG has prepared risk based horizontal internal audit plans for LDAs and SDAs.

Based on a comparative analysis with other jurisdictions, the governments of Australia and the United Kingdom, unlike the Government of Canada, do not have a central internal audit function that conducts horizontal audits or any other types of audits of smaller departments and agencies. These departments and agencies are responsible for determining the required extent and nature of internal audit. In some cases, a larger department provides internal audit services for the smaller agencies with which it is affiliated.

As part of the evaluation, the internal audit practices of the Government of Ontario were examined. Although the internal audit services are not identical, the Government of Ontario employs a centralized approach that is similar to that of the Government of Canada in terms of providing centralized internal audit services to smaller departments and agencies. In 1998, the Government of Ontario approved the restructuring of internal audit in the Ontario public service into one geographically decentralized Internal Audit Division. Currently, there are approximately 220 staff in 11 audit service teams that are organized into client-focused portfolios. Some audit teams provide internal audit services to only one large ministry; other audit teams provide services to a number of smaller ministries that have similar objectives and activities. There is also an enterprise-wide group that performs horizontal audits and risk assessments of common business processes. All internal audit staff report to the Office of the Auditor General of Ontario but generally reside within the ministries that they serve and have an informal reporting relationship with their client chief administrative officers.

As previously stated, the Policy on Internal Audit uses the criteria of $300 million in annual expenditures and 500 FTEs to divide federal government departments and agencies into LDAs and SDAs. Respondents were asked to rate the appropriateness of these criteria, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is very appropriate. As shown in Figure 17, the average overall rating is 3.1, indicating that respondents find the current criteria to be somewhat appropriate.

Figure 17. Appropriateness of Criteria for LDAs and SDAs
Figure 17. Appropriateness of Criteria for LDAs and SDAs. Text version below:
Figure 17. Appropriateness of Criteria for LDAs and SDAs - Text version

The bar graph maps the results of 7 stakeholder groups’ views of the appropriateness of the Policy’s criteria for determining large and small departments and agencies. Scores were as follows: Chief Audit Executives a 3.4, DAC members a 3.3, key informants a 3.2, Deputy Heads of small departments and agencies a 3.1, Chief Financial Officers of small departments and agencies and Deputy Heads of large departments and agencies, 3.0 respectively, and Chief Financial Officers of large departments and agencies 2.9.

The following are the most frequent comments received regarding the appropriateness of the criteria:

  • Three quarters of the SDA CFOs and two thirds of the SDA deputy heads who responded stated that a third category should be employed for micro-agencies and that this third category should have reduced reporting and other requirements for those agencies.
  • Some respondents stated that, in addition to the criteria of 500 FTEs and $300 million in annual expenditures, the level of organizational risk should also be considered in determining whether an organization should be classified as an LDA or SDA.
  • Some representatives of smaller LDAs suggested that the reporting and other requirements for smaller LDAs should be relaxed because these requirements proportionately represent more overhead than for the large audits shops in larger LDAs. In addition, a small agency should not be compared with a large department in terms of internal audit requirements. Other respondents suggested that the threshold should be higher for an LDA, given that the cost of building capacity is high for the entities that are close to the threshold.
  • Some respondents indicated that, for the purposes of MAF, the Treasury Board divides federal government departments and agencies into three categories (i.e., micro-agencies, small departments and large departments). They suggested that all central agencies should use these same three categories, that the thresholds should be consistent across all sectors, and that the obligations and reporting requirements should correspond to the resources allocated to each category.
  • Some respondents stated that the Policy on Internal Audit classifies some organizations as LDAs, whereas other Treasury Board policies (e.g., Policy on Evaluation) classify the same organizations as SDAs.

A review of other Treasury Board policies showed variations in the criteria used to divide organizations into different categories. For example, the criterion used by the 2009 Policy on Evaluation to divide federal government departments and agencies into small and large categories is $300 million in annual expenditures, whereas the criteria used by the Policy on Internal Audit are $300 million in annual expenditures and 500 FTEs. Alternatively, the Guidelines for Chief Financial Officer Qualifications use a three-tiered approach. The MAF rating system also employs a three-tiered approach to divide organizations into large, small and micro-agencies.

Based on a review of departmental websites, the combined expenditures of the 50 SDAs account for less than one per cent of total federal government expenditures. Eight of the 50 SDAs account for about three quarters of all SDA expenditures. Consequently, the remaining 42 SDAs account for only about one quarter of all SDA expenditures and approximately 0.1 per cent of all federal government expenditures.

Conclusion: The Policy has contributed to audit coverage appropriate to the level of risk across government through the provision of accompanying funding to bolster departmental internal audit services, horizontal audits of LDAs and SDAs conducted by the OCG, and the preparation of risk-based audit plans that focus on high-risk areas.

Evaluation Question 12: To what extent has the Policy contributed to strengthening the professionalism of the internal audit function across government?

Respondents were asked to indicate to what extent the internal audit function in their organization has used the improved internal audit standards as a result of the Policy, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. Overall, the respondents surveyed indicated that improved internal audit standards have been used to a significant extent; the average ratings provided by CAEs, DAC members and internal audit staff were 4.3, 3.9 and 3.9, respectively. The following are the most frequent comments provided by respondents:

  • There is more emphasis on meeting Government of Canada internal audit standards.
  • Audits are conducted in a more standardized manner.
  • The Policy has resulted in greater focus on quality assurance.
  • Some respondents indicated that they have initiated external assessments and practice inspections.
  • Some respondents stated that the OCG has developed several useful guides and tools.
  • Some respondents stated that, as a result of clear guidance provided by OCG on internal audit standards and practices, it is possible to build standards and practices into internal audit processes, which helps in training new employees and communicating expectations to internal audit staff.

Government of Canada internal audit standards were developed as part of the implementation of the Policy. The 2001 Policy on Internal Audit stated that internal auditors in the Government of Canada are to use the International Standards for the Professional Practice of Internal Audit produced by the IIA. In addition to recommending the use of these standards, the Government of Canada standards published in conjunction with the implementation of the 2006 Policy on Internal Audit provide other standards such as reporting on internal auditing activities and providing a statement of assurance.

The funding accompanying the 2006 Policy on Internal Audit included approximately $3.9 million per year on an ongoing basis for federal government departments and agencies to be used for the training, certification and professional membership fees of internal audit staff. Based on the cost analysis performed, internal audit training expenditures have increased threefold, from $0.7 million in 2005–06 to $2.1 million in 2009–10, in the 23 LDAs that provided relevant financial information.

The Policy guidelines state that CAEs are expected to have a CIA or other professional accounting designation (Chartered Accountant, Certified General Accountant or Certified Management Accountant) and that, if a CAE has a professional accounting designation, he or she should also obtain a CIA designation. Previous surveys of CAEs conducted by the Treasury Board indicated that there were 12 CAEs who had a CIA designation in 2008, and this number increased to 15 in 2009. As indicated in Table 11, 19 of the 48 CAEs surveyed indicated that they have a CIA. Another 11 CAEs indicated that they are in the process of obtaining a CIA, which, when completed, will mean that approximately 62 per cent of CAEs have a CIA designation. Ten of the 18 CAEs who do not have a CIA designation and who are not in the process of obtaining one have an accounting designation. Almost one half (23) of the 48 CAEs surveyed have an accounting designation; seven CAEs have other professional designations (e.g., Certified Information Systems Auditor, Certified Fraud Examiner).

Table 11. CAE Qualifications
CAEs Who Have a CIA Designation CAEs in the Process of Obtaining a CIA Designation CAEs Who Do Not Have a CIA Designation Total
Number of CAEs surveyed 19 11 18 48
CAEs who have an accounting designation 7 6 10 23
CAEs who have a CGAP designation 3 1 0 4
CAEs who have other professional designations 2 2 3 7

The Policy on Internal Audit does not specify that internal audit staff require a CIA or any other designation. However, the number of internal audit staff who have a CIA designation has almost doubled since the Policy was introduced. A survey of federal government internal audit staff (core public service) in 2008, commissioned by the OCG, indicated that approximately 11 per cent of, or 57 of 530, internal auditors occupying internal audit positions had a CIA designation. According to IIA membership lists, 105, or 19 per cent, of a total of 548 federal government employees currently have a CIA designation. In addition, there are another 14 federal government employees who currently have a Certified Government Auditing Professional designation but not a CIA designation.

CAEs and internal audit staff were asked to indicate to what extent the Policy has contributed to establishing a more certified internal audit function in their organization, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is to a great extent. The average rating given by CAEs was 4.1. The following are the most frequent reasons given by CAEs for their ratings:

  • Certification has increased because the Policy has encouraged CAEs to obtain a CIA designation.
  • More internal audit staff are working to obtain a CIA designation than before the Policy was introduced.
  • CAEs are encouraging their staff to obtain a CIA designation.
  • There is more emphasis on recruiting qualified and certified auditors.
  • Funding for training under the Policy has been a major incentive in encouraging staff to obtain professional designations.
  • Some CAEs already have professional designations and should not be required to obtain a CIA designation as well.

The average rating provided by internal audit staff was 3.4, which is lower than the average rating given by CAEs (4.1). Many of the internal audit staff provided similar reasons for the Policy’s contribution, including the funding provided by the Policy for training, greater emphasis on obtaining a CIA designation, and increased focus on recruiting certified and qualified internal audit staff. However, the most frequent reasons provided by internal audit staff for their rating (i.e., 3 or less) are as follows:

  • Some staff are resisting obtaining a CIA designation because it is not required by the Policy.
  • Some staff are resisting obtaining a CIA designation because they already have a professional designation.
  • MAF assessments have had a greater impact on encouraging internal audit staff to obtain a CIA designation.
  • It is difficult to attract staff who have the right credentials.

Some participants in the focus groups involving DAC members, CAEs, internal audit staff and LDA deputy heads indicated that the Policy has contributed to establishing a more certified internal audit function through the provision of funding for training, stronger emphasis on obtaining a CIA designation, and increased focus on recruiting certified and qualified internal audit staff. Some participants indicated the need for more focus on HR planning in order to resolve the current shortage of qualified internal audit staff in the federal government.

Respondents were asked to indicate the appropriateness of the skill sets, qualifications and capabilities of the internal audit staff in their organization, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is very appropriate. As shown in Figure 18, the highest average ratings were provided by ADMs (4.2) and CAEs (4.1); the lowest average rating was provided by key informants (3.3).

Figure 18. Appropriateness of Internal Audit Staff Skill Sets, Qualifications and Capabilities
Figure 18. Appropriateness of Internal Audit Staff Skill Sets, Qualifications and Capabilities . Text version below:
Figure 18. Appropriateness of Internal Audit Staff Skill Sets, Qualifications and Capabilities - Text version

The bar graph indicates respondents’ views on the appropriateness of the skill sets, qualifications and capabilities of the internal audit function within their organization. Assistant Deputy Ministers reported scores of 4.2, Chief Audit Executives noted 4.1, Chief Financial Officers of large departments and agencies indicated 3.7, while key informants had a score of 3.3.

The following are the most frequent responses received from respondents on this topic:

  • The qualifications of most internal audit staff are appropriate.
  • Attracting qualified auditors is a challenge because the pool of candidates is limited.
  • There is a need for more training, particularly for junior auditors.
  • A high turnover rate is a major problem.
  • There is too much reliance on external resources.

The most frequent reasons provided by key informants for their low rating are as follows:

  • There are still not enough internal audit staff who have CIA designations.
  • More emphasis is needed on functional and/or operational knowledge within an internal audit team.
  • There should be greater focus on a wider range of skills, in areas such as information technology, financial management, Policy and HR, within an internal audit team.

In the November 2004 Report of the Auditor General of Canada, Chapter 1, “Internal Audit in Departments and Agencies,” similar insight is provided into the last two observations regarding the qualifications required by internal audit staff. The report states the following:

A broad range of skills is needed for an effective internal audit group. The necessary skills include staff with a professional designation and staff with specialized knowledge and expertise and whose skills correspond to the business side of a department.

Some key informants stated that the increasing professionalism of the internal audit function is evident in external auditors’ increased reliance on internal audit work performed within departments. For example, the 2010 Fall Report of the Auditor General of Canada states the following regarding an audit of the design and implementation of the Economic Action Plan by the federal government:

As part of our audit, we incorporated the audit findings and conclusions of three internal audits into our own audit work. We selected these audits because they were related most closely to our own objectives and subject matter. We were able to rely on the audit work performed by departments in compliance with audit standards of The Canadian Institute of Chartered Accountants. The three audits we relied on were:

  • The Treasury Board of Canada Secretariat: Audit of the Management of Treasury Board Vote 35
  • Human Resources and Skills Development Canada: Audit of Program Eligibility
  • Natural Resources Canada: Audit of Accelerated Infrastructure Program

Conclusion: The Policy has increased the professionalism of the internal audit function by adding rigour to the internal audit standards and practices used by the federal government and by ensuring a more certified internal audit function across government.

Evaluation Question 13: What, if any, are the unintended outcomes of the Policy?

The most frequent responses received from respondents regarding the unintended positive and negative outcomes of the Policy are outlined as follows.

Positive Outcomes
  • The profile, credibility and visibility of internal audit in the federal government have increased significantly.
  • The value of the advice provided has resulted in the engagement of external DAC members in additional areas and/or issues.
  • OCG horizontal audits have led to the sharing of best practices within SDAs.
  • The complexities of government business, as well as the dedication and professionalism of public servants, are appreciated by DAC members from the private sector.
  • CAEs have been promoted to other senior management positions within the department.
Negative Outcomes
  • The reporting burden has increased.
  • The increase in demand for internal audit staff has resulted in an extremely high turnover rate and the movement of internal audit staff between departments.
  • The shortage of qualified candidates has led to the rapid promotion of internal audit staff.

Conclusion: Although the Policy was intended to increase the role of internal audit, the major unintended outcome is the significant extent to which the profile, credibility and visibility of internal audit in the federal government have increased.

C. Performance: Design and Implementation

This section provides the findings and conclusions for the evaluation issues regarding Policy design and implementation.

Evaluation Question 14: Has the Policy been implemented as intended? What challenges, if any, have occurred in implementing the Policy, and how have these challenges been addressed and overcome?

The CAEs and key informants surveyed stated that one of the major aspects of the Policy that has not been implemented is the reconfiguration of the internal audit community. The respondents stated that the lack of an appropriate classification for internal auditors in the federal government continues to cause recruitment and retention challenges across the internal audit community. Some other aspects of the Policy that have not been fully implemented, which were mentioned less frequently by respondents, include practice inspections and external assessments, annual overview reporting, and technological support for selecting and implementing a common audit information management platform in order to standardize internal audit work processes.

As previously indicated, approximately $9.7 million in ongoing funding commencing in 2009–10 was initially approved in conjunction with the Policy. The funding was to pay for anticipated increases in internal auditor salary levels resulting from the reconfiguration of the internal audit community (i.e., obtaining a separate classification for internal auditors in the federal government). However, the documentation review indicated that this project has not been completed and that a separate classification has not yet been created. The OCG is currently working on a Human Resources Management Framework project, in conjunction with the Office of the Chief Human Resources Officer (OCHRO), that is scheduled for completion by 2012.

Internal audit staff and CAEs were asked to rate the importance of having a different classification for internal auditors with regard to increasing the professionalism of the internal audit function in their organization, on a scale of 1 to 5, where 1 is not at all important, 3 is somewhat important, and 5 is very important. A different classification is viewed as important by CAEs and internal audit staff as shown in the average ratings provided by these groups, at 4.0 and 4.3, respectively. The most frequent reasons given by respondents as to why a different classification is important are as follows:

  • It is very difficult to attract professional staff because a conflict occurs with the departmental HR branch during a competition; university degrees and professional certifications are encouraged for internal audit staff, but the government standard for an AS classification is a high school diploma.
  • It is very difficult to attract qualified candidates, and staff do not stay very long in their position because pay rates are not competitive and the AS classification does not recognize the skills required of internal auditors.
  • There is a high turnover rate among internal auditors due to the lack of an appropriate classification and appropriate pay scales commensurate with internal auditors’ education, experience and skills.
  • There are many different classifications for internal auditors, including AS, AU, FI, ES and CS. There are differences not only in the duties performed but also in the compensation levels. A uniform classification will standardize the roles, responsibilities and portability of auditors.
  • Internal auditors perform the same job but are compensated according to different pay scales because of discrepancies in the different classifications (e.g., AS versus FI).
  • With the AS classification, there is no incentive to obtain a CIA designation or any other certification.
  • It is preferred that internal auditors have university degrees and professional designations, yet they do not earn much more than other staff in the AS category who do not require as much formal education.

In order to obtain suitably qualified internal auditors, some internal audit shops use individuals from a number of other classifications, including FI, ES and CS. Figure 19 shows the different classifications currently held by departmental internal auditors based on a survey conducted by the OCG in 2009.

Figure 19. Classifications of Internal Auditors in 2009
Figure 19. Classifications of Internal Auditors in 2009. Text version below:
Figure 19. Classifications of Internal Auditors in 2009 - Text version

The pie graph shows a breakdown of the various classifications of internal auditors across government. In descending order, the AS stream comprises 62% of the audit population; EXs, 17%, FIs, 11%, ES’, 6%, CS’ 3%, and other, 1%.

A case study was conducted to examine the issues related to obtaining a separate classification for internal auditors. The case study involved a review of the available documentation on the OCG’s Internal Audit HR Management Framework project, as well as interviews with representatives of the OCG and OCHRO representatives. The key findings of the case study are as follows:

  • According to OCG representatives, the Internal Audit HR Management Framework project is a three-year initiative and is currently at its midpoint. The project charter states that the work required by the OCG to professionalize the internal audit community will be completed by 2012. In addition to developing an appropriate classification for internal auditors in the federal government, the project will also involve building the necessary tools from generic work descriptions, competencies, recruitment, professional development and learning, and the infrastructure to manage a community. The project charter stipulates that OCHRO is responsible for approving the finalized standard organizational structures and officially validating the generic work descriptions. The third proposed step is to initiate a conversation with the bargaining agent (Public Service Alliance of Canada). The project charter acknowledges that the bargaining agent’s reactions are unknown. Based on the risks and steps indicated in the charter, it will likely take many years before a new classification for internal auditors is put in place.
  • Although the OCG has done some work on reconfiguration, the efforts to obtain a separate classification for internal auditors have stopped because of the Occupational Group Structure Project that is being carried out by OCHRO, focusing on streamlining the number of classification standards in the public service. Discussions with OCHRO representatives revealed that there are alternatives to creating a separate classification for internal auditors that could be put in place in a shorter period of time and that would be more likely to succeed. One alternative approach suggested by OCHRO representatives is to move internal auditors to another occupational group that is a better fit (e.g., the AU group used by the Office of the Auditor General). Another approach suggested by OCHRO representatives is to employ a strategy similar to that used by federal government evaluation departments to obtain qualified evaluation staff.
  • The interviews with key informants and the surveys of CAEs and internal audit staff indicated that opinions differ considerable with regard to the skills and qualifications required by internal auditors. Although the majority of respondents indicated a need for a different classification for internal auditors (i.e., separate from the AS classification) to increase the professionalism of the internal audit function, several respondents stated that a new classification alone will not solve the HR problems and that the HR strategy should account for the fact that an effective internal audit group requires staff who have a broad range of skills. As previously indicated, the Auditor General’s 2004 report states that the necessary skills include internal audit staff who have a professional designation, as well as specialized knowledge and expertise, and whose skills correspond to the business side of a department.

Based on these findings, it is unlikely that a separate classification for internal auditors will be created in the near future. The OCG has stopped its efforts to obtain a separate classification system because of the Occupational Group Structure Project that is looking at streamlining the number of classification standards in the public service. Consequently, a strategy must be developed to obtain a separate classification for internal audit staff within a reasonable period of time.

The following are some of the other challenges related to Policy implementation that were mentioned most frequently by respondents:

  • Deputy heads were initially resistant to the concept of DACs that have external members. As previously indicated, deputy heads now widely accept the value provided by DACs that have external members.
  • There were difficulties in providing a CAE holistic opinion and annual assurance statement. The requirement to provide a holistic opinion as stipulated in the 2006 Policy was subsequently changed to the requirement for annual overview reporting by the CAE on departmental risk management, control and governance processes. The OCG has recently developed a guidance document on annual overview reporting; however, annual overview reporting is not yet carried out on a regular basis.
  • Initially, there was some resistance to the concept of a direct reporting relationship between the CAE and the Deputy Head. As previously indicated, most CAEs now report directly to deputy heads, and most deputy heads accept the value of the advice and perspective provided by CAEs at the management table.

Based on the cost analysis performed, the actual internal audit expenditures in the LDAs surveyed increased by $35.4 million, from $47.2 million in 2005–06 to $82.6 million in 2009–10. This expenditure increase in the 41 LDAs surveyed is greater than the $24.4 million actually provided annually to these LDAs under the Policy for their incremental internal audit expenses. For some LDAs, the increase in internal audit expenditures from 2005–06 to 2009–10 was more than the incremental funding they received. Other LDAs spent less on internal audit than the incremental funding they have received since the Policy was introduced. As indicated in Table 12, 71 per cent of the LDAs surveyed spent more on internal audit than the incremental funding provided to them. However, the remaining 29 per cent spent less than the incremental funding they received in 2009–10. Departmental representatives cited the shortage of qualified internal auditors and budget cuts across the board as reasons for spending less than the incremental funding they received. Seven departments and agencies that spent more than the incremental funding provided to them forecasted further increases in internal audit expenditures; one department that spent less than the incremental funding it received forecasted further increases in its internal audit expenditures.

Table 12. Internal Audit Expenditures Compared With Incremental Funding Received from 2005–06 to 2009–10
Internal Audit Expenditures from 2005–06 to 2009–10 Number of Organizations % of Total
Less than 50 per cent 4 10
50 to 75 per cent 5 12
76 to 99 per cent 3 7
100 to 125 per cent 3 7
126 to 200 per cent 12 29
201 to 300 per cent 10 25
More than 300 per cent 4 10
Total 41 100

CAEs were asked to indicate the appropriateness of the 2009 changes to the Policy, on a scale of 1 to 5, where 1 is not at all appropriate, 3 is somewhat appropriate, and 5 is very appropriate. The average rating given by CAEs was 3.8. Some of the most frequent responses received from CAEs regarding the appropriateness of the 2009 changes to the Policy are as follows:

  • The changes clarified a number of areas that have been the subject of discussion since the Policy came into force in 2006 (e.g., DAC advisory role, removal of holistic opinion, Minister’s involvement with DAC members) without changing the principles of the Policy.
  • The 2009 changes were very welcome because they reflected reality and eliminated some troublesome elements (e.g., holistic opinion).
  • A few CAEs stated that additional clarification is required on annual overview reporting, and they expressed concerns about whether such a system could be implemented.

Conclusion: The Policy has been implemented as intended, but some aspects of the Policy have not been implemented fully.

Evaluation Question 15: To what extent have the activities carried out by the OCG as a result of the Policy been appropriate for the Policy’s effective implementation?

Respondents were asked to indicate how useful OCG activities and outputs were in helping their organization implement the Policy, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is very useful. As shown in Figure 20, the average ratings provided by the different respondent groups were very similar and ranged from 3.7 (LDA deputy heads) to 3.9 (DAC members and LDA CFOs).

Figure 20. Usefulness of OCG Activities in Policy Implementation
Figure 20. Usefulness of OCG Activities in Policy Implementation. Text version below:
Figure 20. Usefulness of OCG Activities in Policy Implementation - Text version

Five stakeholder groups provided insights on the usefulness of the activities of the Office of the Comptroller General. The bar graph notes that the scores from DAC members and Chief Financial Officers of Large departments and agencies were slightly higher than other groups, at 3.9; both the Deputy Heads and Chief Financial Officers of small departments and agencies reported 3.8; and scores from the Deputy Heads of large departments and agencies were 3.7.

The most frequent reasons given by respondents for their ratings are as follows:

  • OCG activities regarding the recruitment, orientation and training of DAC members have been effective.
  • The best value in DAC workshops is the sharing of experiences.
  • OCG community building activities for CAEs have been very useful.
  • OCG professional practice tools have been useful, but some took too long to be developed.
  • Horizontal audits conducted by the OCG are very useful to SDAs and allow for the sharing of best practices.

The CAEs and internal audit staff that were surveyed were also asked to indicate the usefulness of specific OCG activities and outputs in helping their organization implement the Policy, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is very useful. As shown in Table 13, the three OCG activities rated as most useful were DAC training and networking, guidance on internal audit standards and practices, and guidance to CAEs. The three OCG activities rated as least useful by both the CAEs and internal audit staff were HR strategy and staffing, omnibus supply arrangement (PASS), and horizontal audits of LDAs.

Table 13. Usefulness of OCG Activities in Policy Implementation
Average Rating
Activity CAEs Internal Audit Staff
DAC training and networking 4.1 n/a
Guidance on internal audit standards and practices (e.g., OCG Internal Audit Reference Centre, Government of Canada Internal Auditing Standards) 3.7 3.6
Guidance to CAEs (e.g., CAE coaching and mentoring, annual overview reporting) 3.5 n/a
Development of new internal audit tools and work processes 3.4 3.2
Training and certification of internal auditors 3.2 3.4
Omnibus supply arrangement (PASS) to provide departments with access to contracted audit services 3.1 3.0
Horizontal audits of LDAs 3.0 3.1
HR strategy and staffing (e.g., work descriptions, competency profiles, recruitment drives) 2.9 2.8

The following are the most frequent reasons given for the ratings shown in Table 13:

DAC training and networking
  • The quality of the guidance and support provided to DACs by the OCG has been excellent.
  • The DAC guidebook, training sessions, symposiums and general accessibility and advice from the OCG have been very useful and valuable.
Guidance on internal audit standards and practices
  • The guidance provided by the OCG is useful, but has not been timely; however, this has improved recently.
Guidance to CAEs
  • Coaching and mentoring have been useful, particularly for junior executives.
  • Guidance has been useful but not very timely on some matters (e.g., holistic opinion); however, this has improved recently.
Development of new internal audit tools and work processes
  • The OCG has developed some useful tools but they have not been made available in a timely manner; consequently, some departments have developed their own tools.
Training and certification of internal auditors
  • Training provided by the OCG has been useful, particularly for junior auditors.
  • Some respondents stated that the OCG does not need to be involved directly in training because there are other service providers
Omnibus supply arrangement (PASS) to provide departments with access to contracted audit services
  • PASS was initially time consuming and difficult to use, but has improved recently
Horizontal audits of LDAs
  • These are a useful method for sharing best practices and comparing departmental performance.
  • They should be better linked with the departmental internal audit planning cycle (e.g., the OCG does not give departments enough notice regarding audits it wants to implement).
  • The audits are too broad and the scope is too high level.
  • They help increase coverage in areas not covered by departmental internal audits.
Human resources strategy and staffing
  • The Internal Auditor Recruitment and Development (IARD) Program is valuable.
  • Recruitment drives have not been as effective as anticipated.
  • There has been a lack of progress made toward obtaining a different classification for internal auditors.

The comments received from the focus groups involving DAC members, LDA deputy heads, CAEs and internal audit staff regarding the usefulness of OCG activities and outputs in implementing the Policy are very similar to those indicated here.

Conclusion: The activities carried out by the OCG to help organizations implement the Policy have been appropriate for the effective implementation of the Policy, but some have not been very timely; however, recent improvements have been made.

Evaluation Question 16: How appropriate are the practices and mechanisms that have been put in place by the OCG to monitor the effectiveness and impacts of the activities related to Policy implementation?

The MAF rating system is the key method employed by the OCG to monitor the effectiveness and impacts of the activities related to Policy implementation. This system rates the performance of the internal audit function within LDAs. Respondents were asked to indicate the usefulness of the MAF rating system, on a scale of 1 to 5, where 1 is not at all, 3 is somewhat, and 5 is very useful. As shown in the Figure 21, the average ratings ranged from 3.5 (LDA deputy heads) to 3.2 (internal audit staff).

Figure 21. Usefulness of MAF Rating System in Measuring the Performance of the Internal Audit Function
Figure 21. Usefulness of MAF Rating System in Measuring the Performance of the Internal Audit Function. Text version below:
Figure 21. Usefulness of MAF Rating System in Measuring the Performance of the Internal Audit Function - Text version

This bar graph shows the perception of the usefulness of MAF scores as a measure of performance across internal audit functions. The Deputy Heads of large departments and agencies indicated 3.5 for their views; Chief Audit Executives reported 3.3; and internal audit staff expressed 3.2.

The following are the most frequent reasons given by respondents for their ratings:

  • MAF provides a useful indication of the direction the OCG wants departments to take in order to comply with the Policy.
  • MAF criteria are improving, and recent efforts have been made to make the criteria more transparent.
  • Although useful, MAF requirements impose a heavy reporting burden, particularly on small LDAs that have limited internal audit resources.
  • A significant amount of time is required to document practices solely for MAF purposes.
  • Because of their limited audit experience, MAF assessors sometimes do not fully review or understand what they are assessing.
  • There is too much emphasis on the timeliness of reports versus their quality and scope.

Conclusion: The MAF rating system is the key method employed by the OCG to monitor the effectiveness and impacts of the activities related to Policy implementation. This system is a somewhat useful means of assessing the performance of the internal audit function within LDAs.

Evaluation Question 17: What lessons have been learned as a result of Policy implementation?

The most frequent responses received from CAEs and key informants regarding the lessons that have been learned as a result of Policy implementation are as follows:

  • The provision of incremental funding to departments was critical in successfully implementing the Policy.
  • Sufficient time must be allowed to implement the Policy, particularly if major changes are required.
  • The engagement of deputy heads and the appointment of DAC members must be timely in order to set the direction of internal audit.
  • Initial guidance from the OCG is needed upon introduction of the Policy in order to facilitate implementation, and ongoing guidance is required throughout the entire implementation period.
  • An interim review of Policy implementation must be carried out in order to make necessary adjustments that reflect how the Policy is actually being implemented and to address any challenges occurred in implementation.
  • Regular communication with the internal audit community is required throughout the entire implementation period.
  • The HR implications of the Policy need to be considered and addressed in order to ensure that the Policy can be fully implemented.

Conclusion: The key lessons learned from successful Policy implementation are that a significant investment must be made and a realistic time period must be provided for the implementation of policies and initiatives requiring major system improvements.

D. Performance: Economy and Efficiency

This section provides the findings and conclusions for the evaluation issues regarding the economy and efficiency of policy implementation.

Evaluation Question 18: To what extent are Policy implementation resource levels appropriate?

The November 2004 Report of the Auditor General of Canada, Chapter 1, “Internal Audit in Departments and Agencies,” states that, from 1992–93 to 1999–2000, the number of internal audit staff working in federal government departments and agencies decreased sharply. The number of staff devoted to the internal audit function, including the administrative staff who support the DAC and CAE, increased by 68 per cent, from 359 FTEs in 2005–06 to 602 FTEs in 2009–10, in the 41 organizations that provided financial information. As previously indicated, a survey conducted by Statistics Canada on behalf of OCG Capacity Building and Community Development revealed that the number of internal audit staff in the federal government (core public service administrative staff) has more than doubled, from 190 in 2005 to 479 in 2010.

Based on a comparison with other jurisdictions, the ratio between the number of internal auditors and government expenditures in the Government of Canada is similar to the ratios in the governments of the United Kingdom and Ontario, as shown in Table 14.

Table 14. Ratio of Internal Auditors to Government Expenditures in 2009–10
Canada United Kingdom Ontario
Number of internal auditors 479 2,000 229
2009–10 expenditures ($ billions) 237.8 1,023.5(CAD) 106.3
Number of internal auditors per $ billion expenditures 2.01 1.95 2.15

According to the most recent Global Audit Information Network data produced by the IIA, the average revenues per internal auditor are $324,400 in the private sector. As a comparison, the average expenditures per internal auditor in the federal government are approximately $488,000.

As previously indicated, a survey of LDA deputy heads and other respondents indicated that the Policy has contributed to audit coverage appropriate to the level of risk across government. The respondent groups that gave the highest ratings were CAEs (4.1), LDA deputy heads (4.0), DAC members (3.9) and key informants (3.9); the respondent groups that gave the lowest ratings were SDA deputy heads (3.1) and SDA CFOs (3.1).

CAEs, DAC members and internal audit staff were also asked to rate the appropriateness of Policy implementation resource levels (i.e., comparison of resources provided to resources required) in their organization, on a scale of 1 to 5, where 1 is not at all appropriate, 3 is somewhat appropriate and 5 is very appropriate. DAC members and CAEs provided the highest average rating (3.6); the average rating given by internal audit staff was lower (3.2). The most frequent reason given by internal audit staff for the lower ratings was their inability to obtain sufficient numbers of internal audit staff because of difficulties experienced in hiring qualified internal auditors.

Conclusion: Policy implementation resource levels are appropriate due to the approval of ongoing accompanying funding to implement the Policy.

Evaluation Question 19: To what extent has the Policy been implemented efficiently and economically? Are there more effective ways to implement the Policy or achieve its objectives?

As previously indicated, there is widespread satisfaction with the Policy. Approximately 83 per cent of the LDA deputy heads surveyed stated that they were either “satisfied” or “very satisfied” with the Policy and that one of the most satisfactory elements of the Policy is the establishment of DACs that have external members.

The majority of respondents surveyed did not provide any comments on how to improve the Policy or its implementation, and between one quarter and one third of respondents in each respondent group provided suggestions or recommendations on how to implement the Policy more effectively or achieve its objectives. The most frequent responses received are as follows:

  • Streamline the DAC appointment process because the current process for approving DAC candidates is very time consuming and unpredictable.
  • Accelerate the reclassification of internal auditors because the current classification contributes to internal audit staff recruitment and retention challenges in the federal government.
  • Include the level of risk as a criterion for determining which organizations should be classified as LDAs and SDAs.
  • Establish a separate category for micro-agencies that has reduced requirements for participating in SDA audits conducted by the OCG.
  • Reduce the current reporting burden, including streamlining MAF reporting requirements, particularly for small LDAs.

Some of the less frequent comments received are as follows:

  • Limit the number of DAC members who are former deputy ministers or assistant deputy ministers.
  • Reduce the duplication and inconsistencies between the Policy on Internal Audit and the Policy on Evaluation.
  • Revisit the expectations for annual overview reporting to ensure that they are realistic and can be implemented.
  • Ensure that OCG direction and guidance is provided in a more timely manner.
  • Reduce the overlap between LDA horizontal and departmental audits.
  • Improve the timing and coordination of LDA horizontal audits between the OCG and departments to ensure that horizontal audits are included in the departmental internal audit planning cycle.
  • Ensure that funding received by departments for internal audit is not used for other departmental activities.
  • Devote more effort to obtaining a sufficient number of qualified internal auditors in the federal government.
  • Provide guidance and clarification on the professional qualifications required by the CAE and internal audit staff (e.g., CIA designation).
  • Replace contracted internal audit services with additional internal audit staff where possible because this results in greater value for money.
  • Report DAC costs to ensure transparency.
  • Do not make any changes to the Policy at this time because it is still evolving.
  • Increase the use of technology in internal audit at all levels.

The focus groups involving DAC members, LDA deputy heads, internal audit staff and CAEs provided a number of similar comments and additional clarification on the best way to implement some of the suggestions made.

A comparison of the Policy with similar policies in other jurisdictions (the United Kingdom, Australia and Ontario) produced the following results:

  • Canada has progressed further than the United Kingdom and Australia in ensuring the independence of the CAE by stipulating that the CAE report directly to the Deputy Head. All CAEs except one in the Government of Canada report directly to the Deputy Head. In Australia, the majority of heads of internal audit have a formal reporting line to the Deputy Head, whereas some reporting relationships in the United Kingdom fall short of direct.
  • Canada, the United Kingdom and Australia have established DACs that have external members in some government departments and agencies. In the United Kingdom, the DAC contains only external members; in Canada, all of the DACs except one have a majority of external members; and, in Australia, less than one third of the DACs have a majority of external members.
  • In the United Kingdom, the DAC chair is always an external member; in Canada, 65 per cent of DAC chairs are external; and, in Australia, about one half of DAC chairs are external members. These findings support the Policy, which states that the preferred model is for the DAC chair to be an external member.
  • In the United Kingdom, external DAC members can be retired accounting officers, but not from the department they served. According to a representative of the HM Treasury in the United Kingdom, the appointment of retired accounting officers to DACs is an infrequent occurrence. This finding differs from the current situation in Canada, where 42 per cent of the current external DAC members are former senior federal public servants.
  • In the United Kingdom, the DAC and the internal audit function of large government departments are sometimes responsible for also providing internal audit services for the smaller agencies affiliated with the large government department.

Conclusion: The Policy has been implemented efficiently and economically, but there appear to be opportunities to enhance the Policy and its implementation.

IV. Key Conclusions and Recommendations

This chapter summarizes the key conclusions of the evaluation and provides recommendations to enhance the 2006 Policy on Internal Audit.

A. Key Findings and Conclusions

Overall, the 2006 Policy on Internal Audit has achieved positive results and there is widespread satisfaction with it. The key findings and conclusions of the evaluation of the 2006 Policy on Internal Audit are described as follows.

1. There is a continued need for the Policy.

Although there have not recently been any high-profile breakdowns of control in the federal government, there is still a need for the Policy. The importance of a risk-based internal audit function has increased as a result of the Federal Accountability Act and the new role of deputy heads as the accounting officers of their respective organizations.

In addition, by addressing a number of weaknesses, the Policy has resulted in the following:

  • Independence of departmental audit committees (DACs);
  • Independence of internal audit staff from line management;
  • Sufficient focus on assurance services; and
  • Consistency in internal audit capacity, skills and practices.

2. The Policy has contributed to increasing the independence of the internal audit function from line management across government. The Policy requires DACs to have a majority of external members, as well as a CAE who reports directly to the deputy head of an LDA.

Prior to the Policy, DACs consisted primarily of senior managers from the relevant department. As a result of the Policy, DACs now have external members who are selected for their extensive experience in the public and private sectors and whose goal is to provide objective, independent observations to DACs. In addition, the direct reporting of CAEs to deputy heads and the direct access that CAEs have to the DACs have increased the independence of the internal audit function from line management.

3. The implementation of the Policy, particularly the establishment of DACs that have external members, has contributed to improved risk management, governance, internal control and stewardship of resources in federal government departments and agencies.

The key factors that have helped achieve these intended outcomes include the following:

  • Strong deputy head support for Policy implementation;
  • The establishment of a more professional internal audit function within departments;
  • Improved oversight of the internal audit function by a qualified CAE and DAC members who have audit expertise; and
  • More timely implementation of internal audit recommendations, leading to improved management and business practices.

The Policy’s contribution is important because the Policy is complementary to a number of other federal government policies and initiatives that focus directly on areas such as internal control and risk management.

4. Deputy heads of LDAs have confidence in the assurance provided by the internal audit function and the advice provided by their DAC on risk management, control and governance processes.

Given the depth and breadth of experience held by DAC members, many deputy heads stated that they also use the DAC as a sounding board and a strategic resource for strengthening the overall institution, rather than just the internal audit function.

5. Implementation of the Policy has significantly increased the effectiveness of the internal audit function across government.

The establishment of DACs that have external members has contributed to the effectiveness of the internal audit function within departments. Other contributing factors include the following:

  • The establishment of a CAE position reporting directly to the deputy head;
  • Strong deputy head support for Policy implementation;
  • Incremental funding for departmental internal audit activities;
  • Implementation of Government of Canada audit standards; and
  • Horizontal audits of small department and agencies (SDAs) conducted by the Office of the Comptroller General (OCG).

The proportion of LDAs that obtained an acceptable or strong Management Accountability Framework (MAF) rating for their internal audit function more than doubled, from 42 per cent in 2005–06 to 85 per cent in 2009–10.

6. Implementation of the Policy has resulted in a considerable increase in management action on internal audit report recommendations due primarily to a concerted focus by DACs and CAEs on following up on the implementation of management action plans.

Since the Policy was introduced, there has been more formal tracking of the implementation of internal audit recommendations by CAEs and internal audit staff. Although difficult to quantify, some benefits that have resulted from the implementation of management actions plans include averted risks, improved operational efficiency, and improved risk management and cost savings.

7. The Policy has contributed to increasing the capacity of the internal audit function across government through the approval of $40 million per year in accompanying funding for departmental and OCG internal audit activities.

With the removal of two budgeted expenditures that have not yet been incurred ($9.7 million per year for increased departmental internal audit salaries projected to occur from reconfiguration of the internal audit community, and $1.3 million per year for technological support), the amount approved on an ongoing basis for other departmental and OCG audit activities is approximately $29.4 million per year. This funding has contributed to an increase of 75 per cent in departmental internal audit expenditures, from $47.2 million in 2005–06 to $82.6 million in 2009–10, in the 41 LDAs that provided financial information. The number of staff devoted to the internal audit function, including the administrative staff who support the DACs and CAEs, has increased by 68 per cent, from 359 FTEs in 2005–06 to 602 FTEs in 2009–10, in the LDAs surveyed.

8. The Policy has contributed to audit coverage appropriate to the level of risk across government through the provision of accompanying funding to bolster departmental internal audit services, horizontal audits of LDAs and SDAs conducted by the OCG, and the preparation of risk-based audit plans that focus on high risk areas.

There are drawbacks to the current criteria for determining whether an organization should be classified as an LDA or an SDA (i.e., 500 FTEs and $300 million in annual expenditures). Some of these drawbacks include the following:

  • The criteria do not reflect the characteristics of micro-agencies, which are very small (they account for less than 0.1 per cent of federal government expenditures) and have difficulty meeting the current reporting and other requirements for SDAs.
  • The criteria do not take into account the level of risk within the organization in determining whether an organization should be classified as an LDA or SDA.
  • The criteria do not align with the criteria used in other Treasury Board policies to separate federal government departments and agencies into large and small organizations.

9. The Policy has increased the professionalism of the internal audit function by adding rigour to the internal audit standards and practices used by the federal government and by ensuring a more certified internal audit function across government.

The Policy has led to a stronger emphasis on meeting Government of Canada internal audit standards. Training expenditures have increased threefold, from $0.7 million in 2005–06 to $2.1 million in 2009–10, in the 23 LDAs that provided information on their training expenditures. The number of CAEs who have a Certified Internal Auditor (CIA) designation has increased from 12 in 2008 to 19 in 2010; another 11 CAEs are in the process of obtaining a CIA, which, when completed, will mean that approximately 62 per cent of CAEs have a CIA designation. The number of internal audit staff in the federal government who have a CIA designation has almost doubled, from 57 in 2008 to 105 in 2010.

10. The Policy has been implemented as intended, but some aspects have not been implemented fully.

Some aspects of the Policy that have not been fully implemented are:

  • The reconfiguration of the internal audit community (including the reclassification of internal audit staff);
  • Practice inspections and external assessments; and
  • Annual overview reporting, and technological support for standardizing internal audit work processes.

As previously indicated, the $35.4 million increase in internal audit expenditures from 2005–06 to 2009–10 in the 41 LDAs surveyed is greater than the $24 million provided annually under the Policy to these LDAs for their incremental internal audit expenses. Approximately 71 per cent of the LDAs surveyed spent more on their internal audit function than the incremental funding provided to them; the remaining 29 per cent spent less than the incremental funding they received in 2009–10.

The 2009 changes to the Policy were appropriate and have facilitated implementation of the Policy. The changes clarified a number of areas that have been the subject of discussion since the Policy came into force in 2006 (e.g., DAC advisory role, removal of holistic opinion, Minister’s involvement with DAC members) without changing the principles of the Policy. Additional clarification is needed on how to implement the recent guidelines on annual overview reporting.

11. The activities carried out by the OCG to help organizations implement the Policy have been appropriate for the effective implementation of the Policy, but some have not been very timely; however, recent improvements have been made.

The three OCG activities rated as most useful by CAEs and internal audit staff are DAC training and networking, guidance on internal audit standards and practices, and guidance provided to CAEs. The three activities rated as least useful by both CAEs and internal audit staff are HR strategy and staffing, omnibus supply arrangements (PASS), and horizontal audits of LDAs.

12. The MAF rating system is the key method employed by the OCG to monitor the effectiveness and impacts of the activities related the Policy. Respondents indicated that this system is a somewhat useful means of assessing the performance of the internal audit function within LDAs.

On average, LDA deputy heads, CAEs and internal audit staff stated that MAF ratings are between “somewhat useful” and “useful” as a measure of the performance of their departmental internal audit function. Some respondents stated that MAF criteria have improved recently. Several respondents stated that MAF requirements impose a heavy reporting burden, particularly on small LDAs.

13. Policy implementation resource levels are appropriate due to the approval of ongoing accompanying funding to implement the Policy.

This accompanying funding has enabled federal government departments and agencies to restore the capacity of their internal audit function, which was reduced in the 1990s. Based on a comparison with other jurisdictions, the ratio between the number of internal auditors and government expenditures in the Government of Canada is similar to the ratio in the governments of the United Kingdom and Ontario.

14. The Policy has been implemented efficiently and economically, but there appear to be opportunities to enhance the Policy and its implementation.

Further opportunities to enhance the Policy and its implementation are outlined as follows.

  1. Streamline the DAC appointment process and reduce its unpredictability.

It is necessary to reduce the excessive delays and the unpredictability of the process for obtaining ministerial approval of an external DAC member once a prospective member is selected and agreed upon by the Comptroller General and the Deputy Head. The DAC appointment process must be streamlined because a number of current appointments to DACs will be expiring soon. The renewal of existing DAC members and the appointment of new members need to be carried out quickly to avoid any vacancies in DACs, which would limit the contributions that DAC members are making. In addition, the unpredictability of the process needs to be minimized so that the most qualified DAC candidates are approved and highly qualified DAC candidates are not dissuaded from seeking a DAC membership. One option for streamlining the process and reducing its unpredictability is to delegate Treasury Board ministerial approval to the Secretary of the Treasury Board.

  1. Resolve issues with the current classification of internal auditors in the federal government and ensure that departments are able to staff their internal audit function with a broad range of skills.

One of the intended outcomes of the Policy is to increase the capacity and strengthen the professionalism of the internal audit function within departments. The evaluation provides evidence that the Policy indeed led to the development of a more professional internal audit function within departments as a result of adopting the IIA Professional Practices Framework and encouraging auditors to obtain a professional certification (e.g., a CIA designation). Training expenditures and the number of auditors who have CIA designations have both increased.

To meet the requirements of the Policy on Internal Audit and to address the decrease in the number of internal audit staff that occurred in the 1990s, more internal auditors were needed. The number of internal audit staff in the core public service was 190 in 2005 compared with 479 in 2010. This increase in demand for internal auditors has made it more difficult to attract sufficient internal audit staff. The evaluation also found that CAEs and internal audit staff believe that there is a need for a different classification for internal auditors in order to increase the professionalism of the internal audit function. In particular, CAEs and internal audit staff indicated that educational requirements and compensation levels within the AS category are factors contributing to the difficulties in attracting and retaining qualified internal audit staff.

The evaluation found that there is a shortage of qualified and experienced internal auditors in the federal government and that attracting and retaining qualified internal audit staff remains a challenge. Evidence indicated that the lack of qualified, experienced auditors negatively affected audit coverage and the quality of the work performed. To increase audit coverage and quality, the issues related to attraction and retention need to be resolved.

When addressing the issues related to attracting and retaining internal auditors, the solution must be consistent with the policy direction on professionalizing the internal audit community. In addition, the solution must take into account the concerns expressed by internal auditors and CAEs about the differences in compensation and educational requirements for the various classifications used for the internal audit function. Finally, the solution must include an HR plan that reflects the need for individuals who have a broad range of skills and experience in order to create the balance required for an effective internal audit group. HR planning is required to ensure a common understanding of the duties of the internal audit function and the core competencies needed to carry them out. This planning will support consistent work definitions for internal auditors and enable departments to better balance their HR plans to ensure that their internal audit staff have the desired combination of skills. Consistency in HR planning, work definitions and compensation for auditors will help resolve the recruitment and retention challenges noted in the evaluation.

  1. Investigate alternatives to the current method of dividing federal government departments and agencies into LDAs and SDAs.

It is necessary to determine whether there are alternatives or supplemental approaches that would better balance Policy requirements with the level of risk and the capabilities of different sizes and types of federal government departments and agencies. Although not an exhaustive list, some options proposed by respondents that appear worthy of investigation include the following:

  • Create a separate category for micro-agencies that has reduced requirements for participating in OCG horizontal audits to reflect the fact that most SDAs are very small (42 of the 50 SDAs account for only about one quarter of all SDA expenditures and approximately 0.1 per cent of all federal government expenditures). However, it is also necessary to take into consideration that even small organizations can pose a high level of risk to the federal government (e.g., negative press coverage).
  • Add the level of risk to the two existing criteria for determining whether a federal government department or agency should be considered an LDA or an SDA (i.e., $300 million in annual expenditures and 500 FTEs).
  • Align the criteria used by the Policy on Internal Audit with those used by other Treasury Board policies (e.g., Policy on Evaluation).
  • Increase the number of small LDAs that share a CAE and/or DAC, particularly where two LDAs have similar mandates.
  • Encourage some LDAs to share their CAEs and DACs with the SDAs with which they are affiliated and where there is no potential for conflict of interest.
  1. Limit the number of former public servants who are external members of DACs.

The evaluation data indicated concerns related to the proportion of former public servants who serve on DACs, particularly that having too many former public servants on DACs may reduce the potential for fresh and external perspectives. It is ideal for a DAC to have external members who have a mix of private and public sector skills. The most appropriate composition for a typical DAC that has three external members is as follows:

  • One DAC member who has public sector skills;
  • One DAC member who has skills in financial management, accounting or auditing; and
  • One DAC member who has private sector experience in a field related to the operations of the department.

Regardless of whether a DAC member is from the public or private sector, it is also important to ensure that the required skill sets and competencies are covered when selecting the most appropriate members for a particular DAC. It is preferable to have a maximum of one former public servant and a minimum of two private sector representatives serving as external members on each DAC.

  1. Provide guidance on implementing annual overview requirements.

Since the Policy was introduced, expectations have declined regarding the scope and nature of assurance that should be provided to the Deputy Head by the departmental internal audit function. Expectations have changed from the provision of a holistic opinion (2006 Policy) to an annual assurance report (2009 Policy) and most recently to an annual overview report. The OCG should investigate ways to implement annual overview requirements, and additional guidance should be provided to LDAs in this respect.

Recommendations

Because of the many positive effects of the Policy on Internal Audit and the widespread satisfaction with it, there is no impetus to make significant changes to the Policy. The following recommendations deal primarily with modifications that could be made to further increase the Policy’s effectiveness.

It is recommended that the OCG:

  1. Investigate ways to streamline the DAC appointment process and reduce its unpredictability in order to attract and retain qualified DAC members.
  2. Resolve issues regarding the current classification of internal auditors in the federal government and ensure that departments are able to acquire internal audit staff who have a broad range of skills.
  3. Examine alternatives to the current method of dividing federal government departments and agencies into LDAs and SDAs.
  4. Limit the number of former public servants who are external members of DACs to ensure that DACs are composed of members who have a broad range of skills.
  5. Investigate how annual overview requirements can be implemented in order to provide guidance to LDAs

List of Acronyms

ADM
Assistant Deputy Ministers
AS
Administrative Services Classification
AU
Audit Classification
CA
Chartered Accountant
CAE
Chief Audit Executive
CFE
Certified Fraud Examiners
CFO
Chief Financial Officers
CG
Comptroller General
CIA
Certified Internal Auditor
CISA
Certified Information Systems Auditor
CMA
Certified Management Accountant
COBIT
Control Objectives for Information and related Technology
CoCo
Criteria of Control Board
COSO
Committee of Sponsoring Organizations of the Treadway Commission
DAC
Departmental Audit Committee
DH
Deputy Head
DM
Deputy Minister
DPR
Departmental Performance Report
ES
Economist Classification
FAA
Federal Accountability Act
FI
Financial Officer Classification
FMA
Financial Management Act (Australia)
FTE
Full-time Employee
GCAC
Government of Canada Audit Committee
HM
Her Majesty’s (England)
IARD
Internal Auditor Recruitment and Development
IIA
Institute of Internal Auditors
LDA
Large departments and agencies
MAF
Management Accountability Framework
OAG
Office of the Auditor General
OCG
Office of the Comptroller General
OCHRO
Office of the Chief Human Resources Officer
PASS
Professional Audit Support Services
PSAC
Public Service Alliance of Canada
RBAP
Risk-Based Audit Plan
RPP
Report on Plans and Priorities
SDA
Small Departments and Agencies
SDAAC
Small Departments and Agencies Audit Committee
TB
Treasury Board
TBS
Treasury Board of Canada Secretariat

© Her Majesty the Queen in Right of Canada, represented by the President of the Treasury Board, [2012],
[ISBN: 978-0-660-25637-5]

Page details

Date modified: