Evaluation of the Oversight and Enablement of High Risk, Complex IT Enabled Projects

Conclusions

2.1 Program Background and Context

CIOB’s Information Technology Project Oversight and Review Division (CIOB/ITPROD) was established in 2008 subsequent to some significant project failures, most recently in the 1990s. In reports in 2006^{Footnote 2} and 2007,^{Footnote 3} the Office of the Auditor General (OAG) and the Treasury Board of Canada Secretariat, respectively, highlighted the inherent complexity and risks present in the design and implementation of IT-enabled projects.

The 2006 OAG report indicated that the Secretariat needed to strengthen its role in supporting departments in their capacity to manage projects, and to provide more oversight for IT-enabled projects.

The Secretariat’s 2007 report stated that the reasons for project failure fall into three main categories: i) project conception, in which higher-risk designs tend to be favoured; ii) an unsupportive project environment; and iii) an absence of project management expertise.^{Footnote 4} Among the causes identified were unrealistic costs and schedules, flawed approaches that did not adequately identify the projects as business transformation projects, and a tendency to launch large projects all at once in order to avoid subsequent Treasury Board submissions. Failure was also attributed to “an absence of effective oversight.”^{Footnote 5}

In 2008, the Standing Committee on Public Accounts recommended that “the Treasury Board Secretariat direct departments and agencies to provide financial and performance information on information technology projects expected to cost over $10 million.”^{Footnote 6} In response, the Secretariat noted that dollar value alone was not a sufficient criterion for determining the need for the Secretariat’s oversight. It stated that it would “adopt a risk-based approach to meet Parliamentarians’ need for better information on IT projects ... based on departmental capacity, as well as project risk and complexity.”^{Footnote 7} In 2008, the Secretariat created ITPROD within CIOB.

In 2011, when it created Shared Services Canada, the GC transitioned toward an enterprise-wide, whole-of-government approach to managing its information technology assets. Increasingly, departments^{Footnote 8} work horizontally on joint initiatives and enterprise-wide improvement across government, which adds to the complexity of the environment in which CIOB/ITPROD provides oversight.

2.2 CIOB/ITPROD Mandate

CIOB provides strategic direction and leadership in the pursuit of excellence in information management (IM), information technology (IT), security, access to information and privacy, and service modernization across the GC.

In addition, CIOB plays a leadership role in monitoring and improving the oversight of Canada’s major IM and IT investments by ensuring that IT-enabled projects are well planned, well managed and contribute to the government’s strategic direction in areas such as information management, shared services, use of common infrastructure and interoperability.^{Footnote 9} Treasury Board submissions for high-risk, complex IT-enabled projects are referred to CIOB/ITPROD for review by program sectors, which serve as the Secretariat’s primary contact (single window) for departments.

CIOB/ITPROD has a twofold mission. Part of its mission is at the GC level; part of it is at the CIOB level. Its mission at the GC level is “to build a culture of effective IT-enabled project and portfolio management, governance and oversight in the [Government of Canada].” Its mission as a division within CIOB is “to achieve CIOB excellence in enterprise IT-enabled project portfolio management, analysis and reporting.”

2.3 Governance

The oversight and monitoring of high-risk, complex IT-enabled projects has a number of governance components in addition to sectors' involvement in the Treasury Board submission process:

• An assistant deputy minister (ADM)-level Executive Project Oversight Committee (EPOC) serves as the senior advisory committee to the Chief Information Officer. It advises and makes recommendations on key initiatives and plans that advance executive oversight and governance of IT-enabled projects across the GC. Members come from all departments that have projects currently under active oversight. This committee typically meets four times a year.
• A director general (DG)-level EPOC acts as an advisory body to the ADM EPOC for the development of the biannual briefing Treasury Board on the IT-enabled project portfolio. The DG of CIOB/ITPROD chairs this committee, which meets every six months.
• The GC Chief Information Officer reports to Treasury Board ministers on the health and status of the portfolio of IT-enabled projects monitored by CIOB/ITPROD, as well as on the progress made with policy and enablement activities. CIOB/ITPROD consults Treasury Board of Canada Secretariat committees and sectors on the briefing to the Treasury Board on IT-enabled projects in the following order: the Program Directors Group, the project sector assistant secretaries, the program sectors, and finally, the Executive Committee (EXCO). Committees are consulted on the messaging of the overall presentation. Program sectors are given the presentation, the executive summaries and the project observations for factual review and the executive dashboards as supporting documents for their information. EXCO may be consulted in order to engage the Secretary in the process.

2.4 Oversight Process

CIOB/ITPROD provides several types of oversight:

• Point-in-time project oversight, where the goal is to ensure that the conditions for success are present at the time a Treasury Board submission is developed;
• Ongoing project oversight, where the goal is to ensure that effective departmental project oversight and governance practices are consistently put into practice throughout the life of the project; and
• Project portfolio analysis and reporting, where the goal is to ensure that GC IT-enabled project investments are managed effectively as a portfolio and are in line with GC priorities.

Ideally, CIOB/ITPROD oversight begins at project conception and ends with the submission of the project close-out report. However, departments may involve CIOB/ITPROD later in the process when it becomes clear that an IT component may be required. The oversight process runs in parallel with, but independently from, other Secretariat processes such as the submission review process. CIOB/ITPROD provides oversight by reviewing all relevant project documentation, such as business cases and dashboards. It participates in various governance committees (e.g., steering committees, advisory committees, working groups) and provides oversight and advice.

In the case of high-risk, complex IT-enabled projects that need Treasury Board approval, the Secretariat looks for evidence that departments are using effective project management and oversight practices. Evidence includes an explanation/rationale for the investment (normally in the form of a formal business case); department-approved project documents, such as the project charter and management plan; and evidence of governance structures that use dashboards, gates and independent reviews.^{Footnote 10}

2.5 Oversight Project Portfolio

IT-enabled projects fall into three spheres.

Sphere 1 projects are all IT-enabled projects in government.

Sphere 2 projects are those IT-enabled projects that require Treasury Board approval through the submission process (i.e., they exceed the departmental delegation). As of November 2012, 100 projects were identified as IT-enabled projects^{Footnote 11} falling under Sphere 2.

Sphere 3 projects are those that require a Treasury Board submission and are subject to ongoing oversight by CIOB/ITPROD because the Secretariat deems them to be high-risk and complex. As of November 2012, there were 23 projects in Sphere 3.

Projects may fall under Sphere 3 because they:

• Have an estimated cost of $100 million and over. High-risk projects that are below this cost may be selected for oversight because of the project risk, department's organizational project management capacity assessment, or the nature of the project;
• Include multi-stakeholder governance and/or delivery beyond individual departments;
• Demonstrate a significant change in performance; and/or
• Are deemed to require oversight, at the discretion of ministers.

An illustration of the three spheres of the CIOB/ITPROD's IT-enabled project model is presented in Figure 1.

Figure 1 - Text version

(Figure 1, Page 10, Evaluation of CIOB/ITPROD’s Oversight and Enablement of High-Risk, Complex IT-Enabled Projects Report)

This figure shows CIOB/ITPROD's IT-enabled project model, which consists of three spheres. Sphere 1 refers to all government IT projects. Sphere 2 is a subset of all government IT-enabled projects and refers to all government IT-enabled projects that require Treasury Board approval. Sphere 3 is a subset of Sphere 2 and refers to the Treasury Board of Canada Secretariat's IT-Enabled Project Portfolio, subject to ongoing oversight.

Source: IT enabled Project Portfolio Briefing.

Much of CIOB/ITPROD’s work supports the effective implementation of two Treasury Board policies: the Policy on the Management of Projects and the Policy on Investment Planning – Assets and Acquired Services. The objective of the Policy on the Management of Projects is "...to ensure that the appropriate systems, processes and controls for managing projects are in place, at a departmental, horizontal or government-wide level, and support the achievement of project and program outcomes while limiting the risk to stakeholders and taxpayers." The Policy on Investment Planning – Assets and Acquired Services provides a government-wide approach to the planning of investments in assets and acquired services and provides deputy heads with additional relevant information needed to support investment decisions.^{Footnote 13}

CIOB/ITPROD’s role in project governance and oversight (see Table 1 for definitions) is designed to support project success; however, departments remain responsible for the overall achievement of business outcomes. In other words, CIOB/ITPROD’s role is not to “police” departmental compliance but rather to apply selective oversight.^{Footnote 14} As noted in the Policy on the Management of Projects, deputy heads are ultimately responsible for ensuring that “[a] department-wide governance and oversight mechanism is in place, documented and maintained.”

2.6 Logic Model

The logic model (see Figure 2) is a visual representation of the oversight and enablement of high-risk, complex IT-enabled projects. It identifies how the activities and outputs relate to the short-term, intermediate and long-term outcomes, and to the Secretariat's strategic outcome.

As shown in the logic model, two main categories of activities are undertaken:

1. Active oversight,^{Footnote 15} support of new and ongoing projects and reporting

   CIOB/ITPROD helps determine in which sphere a project falls, conducts due diligence assessments, briefs the Chief Information Officer (CIO), participates in project governance (such as senior review boards), reviews and gives feedback on dashboards, and gets CIO sign-off on projects going to the Treasury Board. These activities focus on the following:

   1. Project concept – The project makes business and technological sense (the business component is shared with program areas of the Secretariat).
   2. Project approach – The mechanics, environment, plan, and the capability of the department to execute the plan.
   3. Business alignment – The processes are in place to ensure that the project stays aligned with business goals, and that the IT-enabled projects are carried out in a systematic way to include a business case with requirements specification, project initiation/planning, project execution, and project close-out.
   4. Implementation – The plans and capability are in place to implement the system, including the project's interactions between departmental IT and business units (i.e., the project sponsor or client).
2. Policy, enablement and outreach
   1. Policy Enablement^{Footnote 16} – The process by which CIOB/ITPROD updates relevant guides and tools such as A Guide to Project Gating for IT-Enabled Projects and the Guide to Executive Project Dashboards.
   2. Ministerial and Parliamentary Reporting – CIOB/ITPROD manages the overall portfolio briefing development and approval processes, including following up on OAG recommendations.
   3. Community Outreach – To sustain the growth of capacity in departments for enhanced project management, CIOB/ITPROD's works to maintain committees such as the Executive Project Oversight Committee (EPOC), the Project Management Advisory Committee (PMAC), and the Chief Information Officer Council.

   The short-term outcomes that are expected from the outputs are as follows:

   • Increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects.
   • Increased awareness across the GC of, and increased capability to apply, enabling tools, policy guidance and best practices in the management, governance and oversight of IT-enabled projects.

   In achieving the short-term outcomes, the following intermediate outcomes are anticipated:

   • Senior management and ministers are enabled to make timely, evidence-based decisions and project course corrections as required; and develop effective responses to risks and issues.
   • Sound management, governance and oversight throughout the lifecycle of a GC IT-enabled project contributes to ensuring value for money.
   • Enabling tools, policy guidance and best practices are applied across the GC in the management, governance and oversight of IT-enabled projects.

   CIOB/ITPROD expects that the achievement of the intermediate outcomes will lead to the following long-term outcomes:

   • Strong and active governance and oversight contributes to the achievement of intended business outcomes of IT-enabled projects.
   • Strategies for sound management, governance and oversight of IT-enabled projects are developed, implemented and sustained throughout the GC.

   Table1.Definitions^{Footnote 17}

   Term	Descriptions
   Oversight	Point-in-time project review: CIOB/ITPROD reviews Treasury Board submissions for project management and governance considerations, and coordinates and consolidates input from other CIOB policy centres, to ensure that conditions for success are present at the time of the submission. Ongoing program sector and departmental engagements: CIOB/ITPROD collaborates with Treasury Board of Canada Secretariat program sectors in providing assistance to departments with the preparation of Treasury Board submissions. This sub‑activity includes the onboarding of new projects into CIOB/ITPROD’s ongoing monitoring and biannual Treasury Board briefing processes. Ongoing project monitoring: CIOB/ITPROD identifies and monitors high‑risk, complex IT‑enabled projects.
   Policy advice	Guidance that CIOB/ITPROD provides regarding Secretariat ‑recommended best practices for the project management, governance and oversight of IT‑enabled projects.
   Enablement	Engagement of departments and adoption of Secretariat policy instruments and best practices for governance and oversight, and Development and implementation of community engagement and development strategies to strengthen the capacity of project executives and practitioners across the GC.
   Business outcomes	The result(s) of a project that address the initial business problem the project was designed to resolve.

Figure 2 - Text version

(Figure 2, Page 14, Evaluation of CIOB/ITPROD’s Oversight and Enablement of High-Risk, Complex IT-Enabled Projects Report)

This graphic illustrates the results chain of the logic model of the CIOB/ITPROD’s oversight and enablement of high-risk, complex IT-enabled projects. The logic model shows that the long-term outcomes of CIOB/ITPROD are “Strong and active governance and oversight contributes to the achievement of intended business outcomes of IT-enabled projects” and “Strategies for sound management, governance and oversight of IT-enabled projects are developed, implemented and sustained throughout the Government of Canada.” A note indicates that business outcomes in this model refer to benefits realization.

The final outcome aims to contribute to the strategic outcome of the Treasury Board of Canada Secretariat: Government is well managed and accountable, and resources are allocated to achieve results.

The CIOB/ITPROD has three intermediate outcomes that aim to contribute to its final outcomes and are presented as follows: “Senior management and ministers are enabled to make timely, evidence-based decisions and project course corrections, and develop effective responses to risk”; “Sound management, governance and oversight throughout the life cycle of a Government of Canada IT-enabled project contributes to ensuring value for money”; and “Enabling tools, policy guidance and best practices are applied across the Government of Canada in the management, governance and oversight of IT-enabled projects.”

Two short-term outcomes aim to contribute to intermediate outcomes. The first short-term outcomes are increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects. The second short-term outcomes aims at increased awareness across the Government of Canada of, and increased capability to apply, enabling tools, policy guidance and best practices in the management, governance and oversight of IT-enabled projects.

Two output clusters aim to achieve the short-term outcomes. The first outputs cluster encompasses project observations documents, dashboard feedback reports, due diligence templates, IT checklists, recommendations, briefings to senior management and Treasury Board ministers, and ongoing program sector and departmental engagements. The second output cluster encompasses policy advice; enabling tools, policy guidance and best practices; outreach and learning events; and community engagement initiatives.

Two activities aim to achieve the two outputs clusters. The first activity expected to lead the first output cluster comprises active oversight, support of new and ongoing projects, and reporting. The second activity leads to the second outputs cluster and comprises policy, enablement and outreach.

Note: Business outcomes in this model refer to benefits realization.

3.1 Scope

The scope of this evaluation covered the range of activities conducted from CIOB/ITPROD's establishment in 2008 to 2012. The evaluation examined the activities of CIOB/ITPROD as per the logic model in its function of oversight and enablement of high-risk, complex IT-enabled projects (Sphere 3) and its impact on other IT-enabled projects that are approved by Treasury Board but that are not subject to CIOB/ITPROD's oversight (Sphere 2 projects). In other words, this evaluation did not address all IT-enabled projects. Therefore the conclusions can only be applied to projects in Sphere 2 or Sphere 3.

The evaluators recognize that Secretariat program sectors and other policy centres such as the Investment Planning Project Management and Procurement Policy Division of the Office of the Comptroller General are an integral part of the broad oversight function of not only IT-enabled projects in the GC, but of project management across the federal government in general. The scope of the evaluation, however, focusses specifically on the activities of CIOB/ITPROD and the achievement of its intended outcomes. It did not assess the mandate of CIOB/ITPROD.

3.2 Methodology

The evaluation assessed the achievement of the outcomes of the CIOB/ITPROD's oversight and enablement of high-risk, complex IT-enabled projects based on the logic model shown in Figure 2. This logic model was developed by CIOB/ITPROD, with the assistance of the Internal Audit and Evaluation Bureau, and was then approved by the Chief Information Officer. CIOB/ITPROD provided the evaluators with a narrative that describes and defines the logic model components and the terms used in it. The logic model and key documents were the sources used to describe CIOB/ITPROD's role within the Secretariat.

This evaluation was based on six lines of evidence, including both qualitative and quantitative methods, drawn from both primary and secondary data sources. The evaluation relied on the following sources of data:

1. Document Review

   The document review included the following:

   • A literature review and jurisdictional comparison that included an examination of practices in Ontario, the United States, Australia, and the United Kingdom. The literature review was supplemented by interviews with representatives from the United States Office of Management and Budget and the Ontario Government Office of the Corporate Chief Information Officer.
   • A review of 20 Treasury Board submissions randomly selected from the original population of 100 for Sphere 2 projects.
   • A review of seven close-out reports for completed projects. Since these projects were primarily initiated before the Treasury Board Guideline on Governance Mechanisms for IT-enabled Business Projects in the Government of Canada^{Footnote 18} was distributed and before the initiation of the CIOB/ITPROD oversight process, the focus of the review was on whether the projects were completed on schedule and on budget, and whether they achieved their objectives.
2. Interviews

   A total of 58 individuals representing 25 departments were interviewed over the telephone or in person. Interviewees comprised the following:

   • Assistant deputy ministers (ADMs): A total of 19 ADMs were interviewed; 13 ADMs from departments with projects under oversight that includes 1 from a closed-out project, and 6 representing Sphere 2 projects.
   • Directors general (DGs): A total of 14 DGs were interviewed from departments with projects under oversight.
   • Project Management Advisory Committee (PMAC) members: A total of 20 PMAC members were interviewed.
   • The Secretariat's Chief Information Office Branch/IT-enabled Project Oversight Division (CIOB/ITPROD): A total of 5 CIOB/ITPROD members were interviewed in person.

   Over the period covered by this evaluation (2008–2012), CIOB/ITPROD was evolving as a program. This evolution took place as CIOB/ITPROD developed new tools and requirements and as the economic environment changed.

3. Case Studies

   The case study analysis involved the selection of 10 of the 23 projects under oversight from 2012 (some of which closed during 2013). These projects represented approximately 40 per cent of the high-risk, complex IT-enabled projects being monitored at the time by CIOB/ITPROD.

   The case study analysis was based on documentation, as well as on interviews with project and CIOB/ITPROD staff. CIOB/ITPROD provided advice on which departments would best meet the selection criteria. The following criteria were used to select the case studies:

   • Materiality;
   • Governance;
   • Breadth of coverage by department;
   • Type of IT system to be implemented;
   • Level of maturity; and
   • Level of interactions with CIOB/ITPROD.

3.3 Limitations

Substantial turnover among project management staff posed a challenge, especially given that most projects examined as part of the evaluation spanned several years. The lack of institutional memory sometimes impeded evaluators' ability to trace the history of a project and thereby assess CIOB/ITPROD's impact. This was mitigated by additional interviews and by reviewing file documents.

Because performance measurement information was also not available, there were no direct indicators of efficiency and economy. This issue was partially mitigated by reviewing project close-out documents and by conducting additional interviews. The high numbers of interviews helped in identifying clear themes and patterns of evidence.

In spite of these limitations, this evaluation meets all of the required policy standards. The number of lines of evidence for this evaluation, overall, contributed to the evaluators' ability to triangulate findings and ensure a high level of confidence in the conclusions and recommendations.

3.4 Presentation of the Findings

The following quantitative scale is used throughout this report to indicate the relative weight of the responses for each of the respondent groups.

• “All/almost all” – Findings reflect the views and opinions of 90 per cent or more of the interviewees in the group.
• “Large majority” – Findings reflect the views and opinions of at least 75 per cent but less than 90 per cent of interviewees in the group.
• “Majority/most” – Findings reflect the views and opinions of at least 50 per cent but less than 75 per cent of interviewees in the group.
• “Some” – Findings reflect the views and opinions of at least 25 per cent but less than 50 per cent of interviewees in the group.
• “A few” – Findings reflect the views and opinions of at least two respondents but less than 25 per cent of interviewees in the group.

4.1 Relevance

4.2 Performance

Relevance

1. CIOB/ITPROD's oversight and enablement program remains relevant. There is a continued need for CIOB/ITPROD to support IT project management capacity, the development of collaborative management approaches and project conception.
2. CIOB/ITPROD's oversight and enablement program is aligned with Government of Canada (GC) roles, responsibilities and priorities. It is also aligned with the Secretariat's strategic outcome.

Performance: Effectiveness, Efficiency and Economy

1. Effectiveness
   1. CIOB/ITPROD has achieved its short-term outcomes to a great extent.
      • All lines of evidence demonstrated increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects.
      • Departments with IT-enabled projects under CIOB/ITPROD's oversight were aware of the tools and guidelines. They also used and applied these tools in projects not subject to CIOB/ITPROD's oversight.
   2. CIOB/ITPROD has progressed in the achievement of its intermediate outcomes.
      1. Senior managers have a greater ability to make timely, evidence-based decisions and project course corrections. The timeliness of CIOB/ITPROD's feedback and its level of involvement, however, did not always meet departments' needs.
      2. Enabling tools, guidance and best practices are being applied to a large extent in the management, governance and oversight of high-risk, complex IT-enabled projects.
      3. Although independent reviews are useful when done well, there is room for improvement in the following areas:
         • The length of time required to contract a reviewer; and
         • The quality of independent reviews.
   3. The long-term outcomes were achieved to some extent.
      1. CIOB/ITPROD has contributed to putting in place the conditions for the success of high-risk, complex IT-enabled projects. However, there is some evidence that the actual achievement of business outcomes is not consistently monitored and evaluated. As a result, the evaluators could not fully assess the extent of contribution to achievement of business outcomes.
      2. Strategies for sound management and governance of IT-enabled projects are in place to some extent. Progress is still needed in the area of performance measurement.
      3. Although there is some indication that project management expertise may be lacking, particularly for horizontal projects, it is not clear to what extent this is the case. A lack of project management expertise is a risk factor for project failure.
2. Efficiency and Economy

   The CIOB/ITPROD oversight and enablement program appears to function efficiently and economically. However, the lack of performance measurement data and extensive resource reductions make it difficult to draw a clear conclusion on this issue.

3. Oversight Model

   Canada is considered a leader in project management best practices, and its oversight model and enablement of high-risk, complex IT-enabled projects parallels other jurisdictions, with the exception of the United States. Nonetheless, some aspects of other models may be worthy of consideration.

Recommendations

1. That CIOB/ITPROD reassess its model of oversight in relation to its resources. It should then develop a revised logic model and performance measurement strategy.
2. That CIOB/ITPROD, in consultation with the Office of the Comptroller General and other relevant stakeholders, explore options to enhance the ability of federal departments and agencies to monitor and evaluate the business outcomes of their IT-enabled projects. The recommendation does not fall solely within the mandate of CIOB-ITPROD.
3. That CIOB/ITPROD discuss the independent review program with the appropriate stakeholders, with a view to strengthening it.
4. That CIOB/ITPROD consider options to streamline the departmental reporting requirements, with a view to enhancing efficiencies and reducing the reporting burden.

Management Response: Agree

Management Response: Agree

Management Response: Agree

Management Response: Agree