Evaluation of the Oversight and Enablement of High Risk, Complex IT Enabled Projects

Evaluation Report
Approved on September 5, 2014

INTERNAL AUDIT AND EVALUATION BUREAU

Table of Contents

List of Acronyms

ADM
Assistant Deputy Minister
ATIP
Access to Information and Privacy
CIOB
Chief Information Officer Branch
DG
Director General
EPOC
Executive Project Oversight Committee
EXCO
Executive Committee
G&S
Good and Services (all costs except salaries)
GC
Government of Canada
I&IT
Information and Information Technology
ICT
Information and communications technology
IM
Information management
IT
Information technology
ITPROD
Information Technology Project Review and Oversight Division
OAG
Office of the Auditor General
PCRA
Project Complexity and Risk Assessment
PMAC
Project Management Advisory Council

Executive Summary

This report presents the results of the evaluation of the oversight and enablement that the Chief Information Officer Branch (CIOB), through its Information Technology Project Review and Oversight Division (ITPROD), provides for high-risk, complex IT-enabled projects. The evaluation was conducted between2012 and 2014 by the Internal Audit and Evaluation Bureau of the Treasury Board of Canada Secretariat (the Secretariat) with the assistance of Goss Gilroy Inc.

The evaluation was undertaken to address senior management's need for information on the relevance and performance of CIOB/ITPROD's oversight and enablement of high-risk, complex IT-enabled projects.Footnote 1

This study began as a review and was later expanded to become an evaluation. It was conducted in accordance with the Policy on Evaluation and the Directive on the Evaluation Function. The methodology included multiple lines of evidence.

The scope of this evaluation covers the range of activities conducted since CIOB/ITPROD was formed in 2008. Although it focuses on Sphere 3 projects–IT-enabled projects assessed by CIOB/ITPROD as being high-risk and complex–a limited inquiry was made into Sphere 2 projects, which are not under CIOB/ITPROD's oversight but which nevertheless require Treasury Board approval. In total, 20 Treasury Board submissions of Sphere 2 projects were reviewed.

An additional sphere of IT-enabled projects was created after this evaluation was completed and therefore is not included or referenced in this report.

Conclusions

Relevance

  1. CIOB/ITPROD's oversight and enablement program remains relevant. There is a continued need for CIOB/ITPROD to support IT project management capacity, the development of collaborative management approaches and project conception.
  2. CIOB/ITPROD's oversight and enablement program is aligned with Government of Canada (GC) roles, responsibilities and priorities. It is also aligned with the Secretariat's strategic outcome.

Performance: Effectiveness, Efficiency and Economy

  1. Effectiveness
    1. CIOB/ITPROD has achieved its short-term outcomes to a great extent.
      • All lines of evidence demonstrated increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects.
      • Departments with IT-enabled projects under CIOB/ITPROD's oversight were aware of the tools and guidelines. They also used and applied these tools in projects not subject to CIOB/ITPROD's oversight.
    2. CIOB/ITPROD has progressed in the achievement of its intermediate outcomes.
      1. Senior managers have a greater ability to make timely, evidence-based decisions and project course corrections. The timeliness of CIOB/ITPROD's feedback and its level of involvement, however, did not always meet departments' needs.
      2. Enabling tools, guidance and best practices are being applied to a large extent in the management, governance and oversight of high-risk, complex IT-enabled projects.
      3. Although independent reviews are useful when done well, there is room for improvement in the following areas:
        • The length of time required to contract a reviewer; and
        • The quality of independent reviews.
    3. The long-term outcomes were achieved to some extent.
      1. CIOB/ITPROD has contributed to putting in place the conditions for the success of high-risk, complex IT-enabled projects. However, there is some evidence that the actual achievement of business outcomes is not consistently monitored and evaluated. As a result, the evaluators could not fully assess the extent of contribution to achievement of business outcomes.
      2. Strategies for sound management and governance of IT-enabled projects are in place to some extent. Progress is still needed in the area of performance measurement.
      3. Although there is some indication that project management expertise may be lacking, particularly for horizontal projects, it is not clear to what extent this is the case. A lack of project management expertise is a risk factor for project failure.
  2. Efficiency and Economy

    The CIOB/ITPROD oversight and enablement program appears to function efficiently and economically. However, the lack of performance measurement data and extensive resource reductions make it difficult to draw a clear conclusion on this issue.

  3. Oversight Model

    Canada is considered a leader in project management best practices, and its oversight model and enablement of high-risk, complex IT-enabled projects parallels other jurisdictions, with the exception of the United States. Nonetheless, some aspects of other models may be worthy of consideration.

Recommendations

  1. That CIOB/ITPROD reassess its model of oversight in relation to its resources. It should then develop a revised logic model and performance measurement strategy.
  2. That CIOB/ITPROD, in consultation with the Office of the Comptroller General and other relevant stakeholders, explore options to enhance the ability of federal departments and agencies to monitor and evaluate the business outcomes of their IT-enabled projects. The recommendation does not fall solely within the mandate of CIOB-ITPROD.
  3. That CIOB/ITPROD discuss the independent review program with the appropriate stakeholders, with a view to strengthening it.
  4. That CIOB/ITPROD consider options to streamline the departmental reporting requirements, with a view to enhancing efficiencies and reducing the reporting burden.

1.0 Introduction

This report presents the findings, conclusions and recommendations of an evaluation conducted between 2012 and 2014 of CIOB/ITPROD's oversight and enablement of high-risk, complex IT-enabled projects. The evaluation was conducted by the Internal Audit and Evaluation Bureau with the assistance of Goss Gilroy Inc.

2.0 Program Profile

2.1 Program Background and Context

CIOB’s Information Technology Project Oversight and Review Division (CIOB/ITPROD) was established in 2008 subsequent to some significant project failures, most recently in the 1990s. In reports in 2006Footnote 2 and 2007,Footnote 3 the Office of the Auditor General (OAG) and the Treasury Board of Canada Secretariat, respectively, highlighted the inherent complexity and risks present in the design and implementation of IT-enabled projects.

The 2006 OAG report indicated that the Secretariat needed to strengthen its role in supporting departments in their capacity to manage projects, and to provide more oversight for IT-enabled projects.

The Secretariat’s 2007 report stated that the reasons for project failure fall into three main categories: i) project conception, in which higher-risk designs tend to be favoured; ii) an unsupportive project environment; and iii) an absence of project management expertise.Footnote 4 Among the causes identified were unrealistic costs and schedules, flawed approaches that did not adequately identify the projects as business transformation projects, and a tendency to launch large projects all at once in order to avoid subsequent Treasury Board submissions. Failure was also attributed to “an absence of effective oversight.”Footnote 5

In 2008, the Standing Committee on Public Accounts recommended that “the Treasury Board Secretariat direct departments and agencies to provide financial and performance information on information technology projects expected to cost over $10 million.”Footnote 6 In response, the Secretariat noted that dollar value alone was not a sufficient criterion for determining the need for the Secretariat’s oversight. It stated that it would “adopt a risk-based approach to meet Parliamentarians’ need for better information on IT projects ... based on departmental capacity, as well as project risk and complexity.”Footnote 7 In 2008, the Secretariat created ITPROD within CIOB.

In 2011, when it created Shared Services Canada, the GC transitioned toward an enterprise-wide, whole-of-government approach to managing its information technology assets. Increasingly, departmentsFootnote 8 work horizontally on joint initiatives and enterprise-wide improvement across government, which adds to the complexity of the environment in which CIOB/ITPROD provides oversight.

2.2 CIOB/ITPROD Mandate

CIOB provides strategic direction and leadership in the pursuit of excellence in information management (IM), information technology (IT), security, access to information and privacy, and service modernization across the GC.

In addition, CIOB plays a leadership role in monitoring and improving the oversight of Canada’s major IM and IT investments by ensuring that IT-enabled projects are well planned, well managed and contribute to the government’s strategic direction in areas such as information management, shared services, use of common infrastructure and interoperability.Footnote 9 Treasury Board submissions for high-risk, complex IT-enabled projects are referred to CIOB/ITPROD for review by program sectors, which serve as the Secretariat’s primary contact (single window) for departments.

CIOB/ITPROD has a twofold mission. Part of its mission is at the GC level; part of it is at the CIOB level. Its mission at the GC level is “to build a culture of effective IT-enabled project and portfolio management, governance and oversight in the [Government of Canada].” Its mission as a division within CIOB is “to achieve CIOB excellence in enterprise IT-enabled project portfolio management, analysis and reporting.”

2.3 Governance

The oversight and monitoring of high-risk, complex IT-enabled projects has a number of governance components in addition to sectors' involvement in the Treasury Board submission process:

  • An assistant deputy minister (ADM)-level Executive Project Oversight Committee (EPOC) serves as the senior advisory committee to the Chief Information Officer. It advises and makes recommendations on key initiatives and plans that advance executive oversight and governance of IT-enabled projects across the GC. Members come from all departments that have projects currently under active oversight. This committee typically meets four times a year.
  • A director general (DG)-level EPOC acts as an advisory body to the ADM EPOC for the development of the biannual briefing Treasury Board on the IT-enabled project portfolio. The DG of CIOB/ITPROD chairs this committee, which meets every six months.
  • The GC Chief Information Officer reports to Treasury Board ministers on the health and status of the portfolio of IT-enabled projects monitored by CIOB/ITPROD, as well as on the progress made with policy and enablement activities. CIOB/ITPROD consults Treasury Board of Canada Secretariat committees and sectors on the briefing to the Treasury Board on IT-enabled projects in the following order: the Program Directors Group, the project sector assistant secretaries, the program sectors, and finally, the Executive Committee (EXCO). Committees are consulted on the messaging of the overall presentation. Program sectors are given the presentation, the executive summaries and the project observations for factual review and the executive dashboards as supporting documents for their information. EXCO may be consulted in order to engage the Secretary in the process.

2.4 Oversight Process

CIOB/ITPROD provides several types of oversight:

  • Point-in-time project oversight, where the goal is to ensure that the conditions for success are present at the time a Treasury Board submission is developed;
  • Ongoing project oversight, where the goal is to ensure that effective departmental project oversight and governance practices are consistently put into practice throughout the life of the project; and
  • Project portfolio analysis and reporting, where the goal is to ensure that GC IT-enabled project investments are managed effectively as a portfolio and are in line with GC priorities.

Ideally, CIOB/ITPROD oversight begins at project conception and ends with the submission of the project close-out report. However, departments may involve CIOB/ITPROD later in the process when it becomes clear that an IT component may be required. The oversight process runs in parallel with, but independently from, other Secretariat processes such as the submission review process. CIOB/ITPROD provides oversight by reviewing all relevant project documentation, such as business cases and dashboards. It participates in various governance committees (e.g., steering committees, advisory committees, working groups) and provides oversight and advice.

In the case of high-risk, complex IT-enabled projects that need Treasury Board approval, the Secretariat looks for evidence that departments are using effective project management and oversight practices. Evidence includes an explanation/rationale for the investment (normally in the form of a formal business case); department-approved project documents, such as the project charter and management plan; and evidence of governance structures that use dashboards, gates and independent reviews.Footnote 10

2.5 Oversight Project Portfolio

IT-enabled projects fall into three spheres.

Sphere 1 projects are all IT-enabled projects in government.

Sphere 2 projects are those IT-enabled projects that require Treasury Board approval through the submission process (i.e., they exceed the departmental delegation). As of November 2012, 100 projects were identified as IT-enabled projectsFootnote 11 falling under Sphere 2.

Sphere 3 projects are those that require a Treasury Board submission and are subject to ongoing oversight by CIOB/ITPROD because the Secretariat deems them to be high-risk and complex. As of November 2012, there were 23 projects in Sphere 3.

Projects may fall under Sphere 3 because they:

  • Have an estimated cost of $100 million and over. High-risk projects that are below this cost may be selected for oversight because of the project risk, department's organizational project management capacity assessment, or the nature of the project;
  • Include multi-stakeholder governance and/or delivery beyond individual departments;
  • Demonstrate a significant change in performance; and/or
  • Are deemed to require oversight, at the discretion of ministers.

An illustration of the three spheres of the CIOB/ITPROD's IT-enabled project model is presented in Figure 1.

Figure 1: CIOB/ITPROD's IT-Enabled Project ModelFootnote 12
CIOB/ITPROD's IT-Enabled Project Model. Text version below:
Figure 1 - Text version

(Figure 1, Page 10, Evaluation of CIOB/ITPROD’s Oversight and Enablement of High-Risk, Complex IT-Enabled Projects Report)

This figure shows CIOB/ITPROD's IT-enabled project model, which consists of three spheres. Sphere 1 refers to all government IT projects. Sphere 2 is a subset of all government IT-enabled projects and refers to all government IT-enabled projects that require Treasury Board approval. Sphere 3 is a subset of Sphere 2 and refers to the Treasury Board of Canada Secretariat's IT-Enabled Project Portfolio, subject to ongoing oversight.

Source: IT enabled Project Portfolio Briefing.

Much of CIOB/ITPROD’s work supports the effective implementation of two Treasury Board policies: the Policy on the Management of Projects and the Policy on Investment Planning – Assets and Acquired Services. The objective of the Policy on the Management of Projects is "...to ensure that the appropriate systems, processes and controls for managing projects are in place, at a departmental, horizontal or government-wide level, and support the achievement of project and program outcomes while limiting the risk to stakeholders and taxpayers." The Policy on Investment Planning – Assets and Acquired Services provides a government-wide approach to the planning of investments in assets and acquired services and provides deputy heads with additional relevant information needed to support investment decisions.Footnote 13

CIOB/ITPROD’s role in project governance and oversight (see Table 1 for definitions) is designed to support project success; however, departments remain responsible for the overall achievement of business outcomes. In other words, CIOB/ITPROD’s role is not to “police” departmental compliance but rather to apply selective oversight.Footnote 14 As noted in the Policy on the Management of Projects, deputy heads are ultimately responsible for ensuring that “[a] department-wide governance and oversight mechanism is in place, documented and maintained.”

2.6 Logic Model

The logic model (see Figure 2) is a visual representation of the oversight and enablement of high-risk, complex IT-enabled projects. It identifies how the activities and outputs relate to the short-term, intermediate and long-term outcomes, and to the Secretariat's strategic outcome.

As shown in the logic model, two main categories of activities are undertaken:

  1. Active oversight,Footnote 15 support of new and ongoing projects and reporting

    CIOB/ITPROD helps determine in which sphere a project falls, conducts due diligence assessments, briefs the Chief Information Officer (CIO), participates in project governance (such as senior review boards), reviews and gives feedback on dashboards, and gets CIO sign-off on projects going to the Treasury Board. These activities focus on the following:

    1. Project concept – The project makes business and technological sense (the business component is shared with program areas of the Secretariat).
    2. Project approach – The mechanics, environment, plan, and the capability of the department to execute the plan.
    3. Business alignment – The processes are in place to ensure that the project stays aligned with business goals, and that the IT-enabled projects are carried out in a systematic way to include a business case with requirements specification, project initiation/planning, project execution, and project close-out.
    4. Implementation – The plans and capability are in place to implement the system, including the project's interactions between departmental IT and business units (i.e., the project sponsor or client).
  2. Policy, enablement and outreach
    1. Policy EnablementFootnote 16 – The process by which CIOB/ITPROD updates relevant guides and tools such as A Guide to Project Gating for IT-Enabled Projects and the Guide to Executive Project Dashboards.
    2. Ministerial and Parliamentary Reporting – CIOB/ITPROD manages the overall portfolio briefing development and approval processes, including following up on OAG recommendations.
    3. Community Outreach – To sustain the growth of capacity in departments for enhanced project management, CIOB/ITPROD's works to maintain committees such as the Executive Project Oversight Committee (EPOC), the Project Management Advisory Committee (PMAC), and the Chief Information Officer Council.

    The short-term outcomes that are expected from the outputs are as follows:

    • Increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects.
    • Increased awareness across the GC of, and increased capability to apply, enabling tools, policy guidance and best practices in the management, governance and oversight of IT-enabled projects.

    In achieving the short-term outcomes, the following intermediate outcomes are anticipated:

    • Senior management and ministers are enabled to make timely, evidence-based decisions and project course corrections as required; and develop effective responses to risks and issues.
    • Sound management, governance and oversight throughout the lifecycle of a GC IT-enabled project contributes to ensuring value for money.
    • Enabling tools, policy guidance and best practices are applied across the GC in the management, governance and oversight of IT-enabled projects.

    CIOB/ITPROD expects that the achievement of the intermediate outcomes will lead to the following long-term outcomes:

    • Strong and active governance and oversight contributes to the achievement of intended business outcomes of IT-enabled projects.
    • Strategies for sound management, governance and oversight of IT-enabled projects are developed, implemented and sustained throughout the GC.

    Table1.DefinitionsFootnote 17

    Term Descriptions
    Oversight
    • Point-in-time project review: CIOB/ITPROD reviews Treasury Board submissions for project management and governance considerations, and coordinates and consolidates input from other CIOB policy centres, to ensure that conditions for success are present at the time of the submission.
    • Ongoing program sector and departmental engagements: CIOB/ITPROD collaborates with Treasury Board of Canada Secretariat program sectors in providing assistance to departments with the preparation of Treasury Board submissions. This sub‑activity includes the onboarding of new projects into CIOB/ITPROD’s ongoing monitoring and biannual Treasury Board briefing processes.
    • Ongoing project monitoring: CIOB/ITPROD identifies and monitors high‑risk, complex IT‑enabled projects.
    Policy advice
    • Guidance that CIOB/ITPROD provides regarding Secretariat ‑recommended best practices for the project management, governance and oversight of IT‑enabled projects.
    Enablement
    • Engagement of departments and adoption of Secretariat policy instruments and best practices for governance and oversight, and
    • Development and implementation of community engagement and development strategies to strengthen the capacity of project executives and practitioners across the GC.
    Business outcomes
    • The result(s) of a project that address the initial business problem the project was designed to resolve.
Figure 2: Logic Model of CIOB/ITPROD's Oversight and Enablement of High-Risk, Complex IT-Enabled Projects
Logic Model of CIOB/ITPROD's Oversight and Enablement of High-Risk, Complex IT-Enabled  Projects. Text version below:
Figure 2 - Text version

(Figure 2, Page 14, Evaluation of CIOB/ITPROD’s Oversight and Enablement of High-Risk, Complex IT-Enabled Projects Report)

This graphic illustrates the results chain of the logic model of the CIOB/ITPROD’s oversight and enablement of high-risk, complex IT-enabled projects. The logic model shows that the long-term outcomes of CIOB/ITPROD are “Strong and active governance and oversight contributes to the achievement of intended business outcomes of IT-enabled projects” and “Strategies for sound management, governance and oversight of IT-enabled projects are developed, implemented and sustained throughout the Government of Canada.” A note indicates that business outcomes in this model refer to benefits realization.

The final outcome aims to contribute to the strategic outcome of the Treasury Board of Canada Secretariat: Government is well managed and accountable, and resources are allocated to achieve results.

The CIOB/ITPROD has three intermediate outcomes that aim to contribute to its final outcomes and are presented as follows: “Senior management and ministers are enabled to make timely, evidence-based decisions and project course corrections, and develop effective responses to risk”; “Sound management, governance and oversight throughout the life cycle of a Government of Canada IT-enabled project contributes to ensuring value for money”; and “Enabling tools, policy guidance and best practices are applied across the Government of Canada in the management, governance and oversight of IT-enabled projects.”

Two short-term outcomes aim to contribute to intermediate outcomes. The first short-term outcomes are increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects. The second short-term outcomes aims at increased awareness across the Government of Canada of, and increased capability to apply, enabling tools, policy guidance and best practices in the management, governance and oversight of IT-enabled projects.

Two output clusters aim to achieve the short-term outcomes. The first outputs cluster encompasses project observations documents, dashboard feedback reports, due diligence templates, IT checklists, recommendations, briefings to senior management and Treasury Board ministers, and ongoing program sector and departmental engagements. The second output cluster encompasses policy advice; enabling tools, policy guidance and best practices; outreach and learning events; and community engagement initiatives.

Two activities aim to achieve the two outputs clusters. The first activity expected to lead the first output cluster comprises active oversight, support of new and ongoing projects, and reporting. The second activity leads to the second outputs cluster and comprises policy, enablement and outreach.

Note: Business outcomes in this model refer to benefits realization.

3.0 Evaluation Context

The purpose of the evaluation was to assess the relevance and performance of CIOB/ITPROD activities (effectiveness, efficiency and economy) in accordance with the requirements of the Treasury Board Directive on the Evaluation Function (2009). In addition, it examined the design, resourcing, processes, influences and limitations of CIOB/ITPROD. Relevance was examined in terms of the continued need for CIOB/ITPROD, its alignment with federal and departmental roles, responsibilities and priorities and with the Treasury Board of Canada Secretariat's strategic outcome. Performance was assessed in terms of effectiveness, efficiency and economy. Other oversight models were also reviewed.

3.1 Scope

The scope of this evaluation covered the range of activities conducted from CIOB/ITPROD's establishment in 2008 to 2012. The evaluation examined the activities of CIOB/ITPROD as per the logic model in its function of oversight and enablement of high-risk, complex IT-enabled projects (Sphere 3) and its impact on other IT-enabled projects that are approved by Treasury Board but that are not subject to CIOB/ITPROD's oversight (Sphere 2 projects). In other words, this evaluation did not address all IT-enabled projects. Therefore the conclusions can only be applied to projects in Sphere 2 or Sphere 3.

The evaluators recognize that Secretariat program sectors and other policy centres such as the Investment Planning Project Management and Procurement Policy Division of the Office of the Comptroller General are an integral part of the broad oversight function of not only IT-enabled projects in the GC, but of project management across the federal government in general. The scope of the evaluation, however, focusses specifically on the activities of CIOB/ITPROD and the achievement of its intended outcomes. It did not assess the mandate of CIOB/ITPROD.

3.2 Methodology

The evaluation assessed the achievement of the outcomes of the CIOB/ITPROD's oversight and enablement of high-risk, complex IT-enabled projects based on the logic model shown in Figure 2. This logic model was developed by CIOB/ITPROD, with the assistance of the Internal Audit and Evaluation Bureau, and was then approved by the Chief Information Officer. CIOB/ITPROD provided the evaluators with a narrative that describes and defines the logic model components and the terms used in it. The logic model and key documents were the sources used to describe CIOB/ITPROD's role within the Secretariat.

This evaluation was based on six lines of evidence, including both qualitative and quantitative methods, drawn from both primary and secondary data sources. The evaluation relied on the following sources of data:

  1. Document Review

    The document review included the following:

    • A literature review and jurisdictional comparison that included an examination of practices in Ontario, the United States, Australia, and the United Kingdom. The literature review was supplemented by interviews with representatives from the United States Office of Management and Budget and the Ontario Government Office of the Corporate Chief Information Officer.
    • A review of 20 Treasury Board submissions randomly selected from the original population of 100 for Sphere 2 projects.
    • A review of seven close-out reports for completed projects. Since these projects were primarily initiated before the Treasury Board Guideline on Governance Mechanisms for IT-enabled Business Projects in the Government of CanadaFootnote 18 was distributed and before the initiation of the CIOB/ITPROD oversight process, the focus of the review was on whether the projects were completed on schedule and on budget, and whether they achieved their objectives.
  2. Interviews

    A total of 58 individuals representing 25 departments were interviewed over the telephone or in person. Interviewees comprised the following:

    • Assistant deputy ministers (ADMs): A total of 19 ADMs were interviewed; 13 ADMs from departments with projects under oversight that includes 1 from a closed-out project, and 6 representing Sphere 2 projects.
    • Directors general (DGs): A total of 14 DGs were interviewed from departments with projects under oversight.
    • Project Management Advisory Committee (PMAC) members: A total of 20 PMAC members were interviewed.
    • The Secretariat's Chief Information Office Branch/IT-enabled Project Oversight Division (CIOB/ITPROD): A total of 5 CIOB/ITPROD members were interviewed in person.

    Over the period covered by this evaluation (2008–2012), CIOB/ITPROD was evolving as a program. This evolution took place as CIOB/ITPROD developed new tools and requirements and as the economic environment changed.

  3. Case Studies

    The case study analysis involved the selection of 10 of the 23 projects under oversight from 2012 (some of which closed during 2013). These projects represented approximately 40 per cent of the high-risk, complex IT-enabled projects being monitored at the time by CIOB/ITPROD.

    The case study analysis was based on documentation, as well as on interviews with project and CIOB/ITPROD staff. CIOB/ITPROD provided advice on which departments would best meet the selection criteria. The following criteria were used to select the case studies:

    • Materiality;
    • Governance;
    • Breadth of coverage by department;
    • Type of IT system to be implemented;
    • Level of maturity; and
    • Level of interactions with CIOB/ITPROD.

3.3 Limitations

Substantial turnover among project management staff posed a challenge, especially given that most projects examined as part of the evaluation spanned several years. The lack of institutional memory sometimes impeded evaluators' ability to trace the history of a project and thereby assess CIOB/ITPROD's impact. This was mitigated by additional interviews and by reviewing file documents.

Because performance measurement information was also not available, there were no direct indicators of efficiency and economy. This issue was partially mitigated by reviewing project close-out documents and by conducting additional interviews. The high numbers of interviews helped in identifying clear themes and patterns of evidence.

In spite of these limitations, this evaluation meets all of the required policy standards. The number of lines of evidence for this evaluation, overall, contributed to the evaluators' ability to triangulate findings and ensure a high level of confidence in the conclusions and recommendations.

3.4 Presentation of the Findings

The following quantitative scale is used throughout this report to indicate the relative weight of the responses for each of the respondent groups.

  • “All/almost all” – Findings reflect the views and opinions of 90 per cent or more of the interviewees in the group.
  • “Large majority” – Findings reflect the views and opinions of at least 75 per cent but less than 90 per cent of interviewees in the group.
  • “Majority/most” – Findings reflect the views and opinions of at least 50 per cent but less than 75 per cent of interviewees in the group.
  • “Some” – Findings reflect the views and opinions of at least 25 per cent but less than 50 per cent of interviewees in the group.
  • “A few” – Findings reflect the views and opinions of at least two respondents but less than 25 per cent of interviewees in the group.

4.0 Findings

4.1 Relevance

4.1.1 Continued Need for the Program

Evaluation Question: To what extent is there a continuing need for the program?

Findings

CIOB/ITPROD's oversight and enablement program remains relevant. There is a continued needed for CIOB/ITPROD to support IT project management capacity, the development of collaborative management approaches and active oversight.

The evaluation confirmed the ongoing need for CIOB/ITPROD despite advancements in many departments since the program began in 2008. Most interviewees indicated that the CIOB/ITPROD model of oversight and enablement is still appropriate and that the program is still needed in order to help departments build and sustain their project management capacity. Based on an analysis of interviewees' comments, the problems in three key areas (project conception, project environment, and IT qualifications and experience) remain a challenge.

Several case studies demonstrated a need for departments to define project requirements more clearly in the planning phase, in particular, scheduling and cost. This was illustrated by a review of project files that showed that most projects (9 of 10) did not meet the original completion schedule. Just over half were completed in accordance with, or under, their original costing estimates, although several of these projects were re-scoped or de-scoped in doing so. Similarly, the evaluation team's analysis of a selection of project close-out reports shows that almost none of these projects were completed on schedule.

Some additional issues of note were raised in case study interviews:

  • Turnover among project team members or maintaining adequate project capacity over the course of a multi-year assignment is a challenge.
  • CIOB/ITPROD is needed to provide the vision and framework for managing joint projects.
  • There is a need for an oversight body with an arm's length view (horizontal approach).

Some ADMs and a few DGs for the oversight projects who were interviewed felt that CIOB/ITPROD support was still needed to develop a horizontal approach, to support collaborative management on joint projects, and to update guides and tools when sharing accountability. A few interviewees felt that CIOB/ITPROD support was still needed to help departments build and sustain their IT project management capacity. The interviewees indicated that the GC has only a small cadre of project managers who have the necessary capabilities to manage high-risk, complex IT-enabled projects, and that a project management culture is not yet universal in the GC.

4.1.2 Alignment with GC roles and responsibilities, federal priorities and the Treasury Board of Canada Secretariat's strategic outcome

Evaluation Question: To what extent is CIOB/ITPROD aligned with GC roles and responsibilities, federal priorities and the Treasury Board of Canada Secretariat's strategic outcome?

Findings

The CIOB/ITPROD oversight and enablement program is aligned with GC roles and responsibilities, federal priorities and the Secretariat's strategic outcome.

CIOB/ITPROD work objectives are aligned with GC roles and responsibilities, and with the federal priorities of returning to balanced budgets and establishing more efficient government operations. Economic Action Plan 2013 calls for new ways to standardize, consolidate and transform the way government does business, to improve services and achieve efficiencies. The Secretariat’s strategic outcome is “Government is well managed and accountable, and resources are allocated to achieve results.” Through active oversight of IT-enabled projects, the activities of the program speak to the role of government in providing sound stewardship of these investments.

Similarly, CIOB/ITPROD contributes to the Secretariat’s strategic priorities, namely, “Support the government in ensuring value for money” and “Advance initiatives to modernize government operations.” Almost all of the Sphere 3 projects overseen by CIOB/ITPROD, and many of the Sphere 2 projects, are actually business process transformation projects that have IT components. These efforts form an important part of many modernization efforts across government, often generating efficiencies for Canadians or replacing legacy systems.

CIOB/ITPROD’s work is consistent with and supportive of the Policy on the Management of Projects and the Policy on Investment Planning – Assets and Acquired Services, both of which target the achievement of value for money and sound stewardship as key objectives.

Lastly, CIOB/ITPROD’s work is a direct response to the concerns raised in documents by the Standing Committee on Public Accounts and the OAG. Although the OAG has stated that, in some respects, it is too early to see the full impact of CIOB/ITPROD’s work, its 2011 report stated that, “These governance mechanisms that the Secretariat has recently implemented are a significant improvement in monitoring the health of large IT projects in which the government invests billions of dollars.”Footnote 19

4.2 Performance

4.2.1 Effectiveness

Short-Term Outcomes

Evaluation Question: To what extent has CIOB/ITPROD contributed to increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects?

Findings

All lines of evidence demonstrated increased senior management and ministerial awareness of risks, issues and best practices in the management, governance and oversight of IT-enabled projects.

One of the short-term outcomes of CIOB/ITPROD is increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects. The interviews, case studies and document review all indicated that awareness and use of best practices is extensive among senior managers who are involved in high-risk, complex IT-enabled projects, and among ministers. Awareness has been achieved through the following:

  • The dissemination of standardized tools and guidance instruments to departments for improving project management;
  • Meetings between ADMs and CIOB;
  • The establishment of CIOB/ITPROD committees such as EPOC and PMAC;
  • Feedback provided in previous years to departments by CIOB/ITPROD (e.g., on project management, as well as on navigating the Treasury Board requirements for approval);
  • Biannual briefings on Sphere 3 projects to Treasury Board ministers provided by the Chief Information Officer.

PMAC interviewees in particular felt that the increased involvement of senior management has led to the integration of project management practices into departmental culture.

The document review provided some supporting evidence that awareness has increased. Most notably, a 2011 OAG statement indicates that “overall progress on the implementation of risk management for the project we reviewed has been satisfactory,”Footnote 20 and a nomination for the 2012 Public Sector Award of Excellence was based on the increased awareness of departments.

CIOB/ITPROD’s documentation supported the assertion that there has been a sharp increase in awareness of the IT project management guidance and tools that are available as a result of outreach to EPOC ADM members and EPOC DG members, ADM support of dashboards and independent reviews, and dissemination of the 2007 report Improving IT Project Performance: Conception, Assessment and Monitoring. It also indicated that “...whether to adopt these or adapt other leading practices remains a departmental decision.”Footnote 21

Evaluation Question: To what extent has there been increased awareness across the GC of, and increased capability to apply, tools, policy guidance and best practices in the management, governance and oversight of IT-enabled projects?

Findings

Departments with IT-enabled projects under CIOB/ITPROD's oversight were aware of the tools and guidelines. In addition, these tools were used and also applied on other projects not subject to CIOB/ITPROD's oversight.

All 10 case studies of Sphere 3 projects showed evidence of government-wide awareness of tools and guidance. In every project examined, reference materials developed and promoted by CIOB/ITPROD were used. Findings from these case studies indicated that the dashboards were used by all projects and were, in some cases, explicitly mentioned as tools that contributed to effective project governance. In addition, some interviewees noted that the use of dashboards is increasing in projects that are not subject to CIOB/ITPROD's oversight.

A review of Treasury Board submissions for Sphere 2 projects also indicated frequent use of CIOB/ITPROD's tools and best practices even though it was not mandatory. Over 50 per cent of the 20 submissions reviewed included governance structures, risk management plans, business cases and project charters.

Almost everyone interviewed about Sphere 3 projects was familiar with CIOB/ITPROD tools and guidelines but expressed mixed views as to whether it was the program specifically that had raised their awareness. Most of those who indicated this was not the case, explained that their departments already had their own tools (often larger departments).

Intermediate Outcomes

Evaluation Question: To what extent are senior management and ministers able to make timely, evidence-based decisions and project course corrections, and develop effective responses to risks and issues as a result of CIOB/ITPROD's work?

Findings

CIOB/ITPROD tools helped senior managers assess project status, enabling them to make evidence-based decisions and project course corrections. The frequency of CIOB/ITPROD involvement and guidance was mixed, with some interviewees expressing overall positive views and others indicating that the involvement was infrequent or occasional.

The gating system and executive project dashboards are two of the tools used by GC departments and CIOB/ITPROD to inform decision makers about project status. All of the ADMs interviewed found the gating system useful in ensuring that they obtained periodic external assessments of project health, enabling them to make timely course corrections. The gating system was regularly used with the independent review process, which provided ADMs with a third-party opinion that contributed to informed, objective decision making on the project.

Interviewees also noted that executive project dashboards provided sufficient information for decisions on risks and project status. Departments under oversight submitted monthly dashboards to CIOB/ITPROD on project scope, budget and overall risks. This exercise, plus an overall assessment made by CIOB/ITPROD, provided senior management with evidence for project course corrections. Most ADMs and DGs interviewed (with some variation among DGs) indicated that CIOB/ITPROD's oversight was useful to senior management in improving the consistency and quality of reporting on projects, providing examples of best practices, and in helping clarify Treasury Board's information needs. CIOB/ITPROD's involvement also assisted senior management in taking corrective action when needed.

The case studies showed that most departments considered CIOB/ITPROD involvement infrequent, occasional, limited or intermittent. However, CIOB/ITPROD indicated that its involvement varied with risk, because of limited resources. CIOB/ITPROD was more involved in projects that were high-risk, at critical junctures or facing important challenges; it was less involved in projects that were low-risk.

Evaluation Question: To what extent are enabling tools, policy guidance and best practices applied in the management, governance and oversight of IT-enabled projects?

Findings

Enabling tools, guidance and best practices are being applied to a large extent in the management, governance and oversight of high-risk, complex IT-enabled projects. Independent reviews are useful when done well. However, senior management had concerns about the length of time for contracting and the variable quality of the independent reviews.

A number of factors may have influenced the adoption of enabling tools and best practices such as the Policy on the Management of Projects and the Policy on Investment Planning – Assets and Acquired Services to which all departments listed in Schedule II of the Financial Administration Act are subject, unless specific acts or regulations override them.

CIOB/ITPROD enabling tools, guidance and best practices are designed to support sound management, governance and oversight. Interview results and case studies indicated that these tools and guidelines were being applied to a large extent in Sphere 3 projects and to a lesser extent in Sphere 2 projects. The ADM interviews showed that the majority were using the tools and guidelines, and were embedding them with or incorporating them into those of their departments.

The case studies and interviews indicated that there is a growing recognition of the importance of robust governance and well-functioning project management systems. They showed that independent reviews and gating have had a positive impact on the success of projects by identifying critical issues early, thereby avoiding wasted resources and supporting cost-effectiveness.

Case study evidence indicated that CIOB/ITPROD's tools and guidance were used to a significant extent. In all case studies, project charters and business cases had been written, and dashboards were completed monthly and sent to CIOB/ITPROD as required. In most cases, project management plans had been prepared and were available, a project gating methodology had been implemented, and independent reviews had either been completed or were planned.

Several project close-out reports indicated that planning and budgeting for third-party reviews at strategic points in the project lifecycle provided timely, effective and neutral observations from perspectives beyond those involved in day-to-day project management.

CIOB has a supply arrangement for reviewer services, and departments access the list when an independent review is required. However, some ADMsexpressed concern about the length of time required to finalize the contract for a reviewer and about the variable quality of independent reviews. It was suggested that the supply arrangement could better vet potential consultants to ensure that they have adequate training and background, given the importance of the independent review process. CIOB/ITPROD was also concerned about reviews being edited and about key issues being omitted from the statement of work.

Departments with Sphere 2 projects showed significant variation in the application of CIOB/ITPROD tools and guidance, based on the size and complexity of the project, and whether departments had its own pre-existing tools. The submission review of Sphere 2 projects showed that departments with strong existing project management capacity (e.g., Canada Revenue Agency and Public Works and Government Services Canada) had already developed their own tools and were therefore less likely to use those of CIOB/ITPROD. Instead, their interest was in best practices, and aligning their tools with those of CIOB/ITPROD.

Long-Term Outcomes

Evaluation Question: To what extent does strong and active governance and oversight contribute to the achievement of the intended business outcomes of IT-enabled projects?

Findings

CIOB/ITPROD has contributed to putting in place conditions for the success of high-risk, complex IT-enabled projects. However, there is some evidence that the actual achievement of business outcomes is not consistently monitored and evaluated. As a result, the evaluators could not fully assess the extent of contribution to the achievement of business outcomes.

There was no conclusive evidence in the document review of a link between CIOB/ITPROD governance and oversight and improved ability of departments to realize business outcomes. Departments did not appear to uniformly collect performance measurement information that could have demonstrated this. Although many interviewees were of the opinion that project performance for high-risk, complex IT-enabled project has improved since the establishment of CIOB/ITPROD and the new processes for oversight, they could not provide empirical evidence.

The evaluation found that CIOB/ITPROD contributed to putting in place conditions for the success of IT-enabled business projects, including large, complex ones. The conditions for success include having a clear justification, a clear and reasonable scope, a reasonable IT approach, reasonable estimates, clear governance, clear go/no-go gates and independent reviews.Footnote 22 This was demonstrated by CIOB/ITPROD’s work on active governance, the growth of such structures among departments, and the increased use of guidance materials and best practices.

Although other factors have influenced project success (e.g., the Policy on the Management of Projects and the Policy on Investment Planning – Assets and Acquired Services, based on the document review, the success of high-risk, complex IT-enabled projects is also linked to improved governance and oversight. CIOB/ITPROD’s involvement, guidance and mechanisms played a role in business transformation, progress and success.

Evaluation Question: To what extent are strategies for sound management, governance and oversight of IT-enabled projects developed, implemented and sustained throughout the GC?

Findings

Evidence showed that strategies for sound management and governance of IT-enabled projects are in place to some extent. Progress is still needed in performance measurement. There is some indication that project management expertise may be lacking, particularly for horizontal projects. However, it is not clear to what extent this is the case. As previously noted, a lack of project management expertise is a risk factor for project failure.

A second long-term outcome expected by CIOB/ITPROD is government-wide development, implementation and sustainability of strategies for sound management,Footnote 23 governance and oversight of IT-enabled projects. The interviews and case studies indicated that sustainable strategies to foster project performance have, for the most part, been put in place across government.

Most interviewees indicated that CIOB/ITPROD has helped promote and implement good governance structures in departments to foster project performance, and that it has contributed to strengthening the business component of IT-enabled projects. Most DGs said that CIOB/ITPROD's work has helped address the three main sources of problems that cause IT-enabled projects to fail, which were project conception (in which higher-risk designs tended to be favored); an unsupportive project environment; and a lack of project management expertise.

The review of Treasury Board submissions for Sphere 2 projects indicated that 95 per cent of submissions incorporated a proposed governance structure. Examples included commitments to adhere to departmental project management and governance guidelines, commitments to establish a project management office with direct reporting to the Secretariat, and commitments to establish an oversight body (both internal to the department and sometimes with participants outside the department).

The evaluation team noted little evidence of mechanisms for measuring and evaluating the achievement of business outcomes. Although all project charters examined in the case studies indicated the intent to measure the achievement of business outcomes, only some included a measurement plan (e.g., specific indicators, a complete performance measurement strategy). Others identified long-term performance indicators but did not include a measurement plan. Some project close-out reports indicated that performance measurement strategies had been established and were being used to assess the project's effect on business processes; however, there was no evidence that results were being measured. Based on these observations, some performance measurement is being carried out. Without consistent strategies in place to collect the necessary performance data, however, the GC will have difficulty determining whether the business outcomes of its IT-enabled projects are being achieved.

As stated before, sound management of IT-enabled projects refers to management processes and practices based on best practices and generally accepted methodologies, conducted by managers with applicable expertise, skills and competencies.Footnote 24 The 2006 OAG report indicated that the Secretariat needed to strengthen its role in supporting departments in their capacity to manage projects. The Secretariat's 2007 report, Improving IT Project Performance: Conception, Assessment and Monitoring, cited the absence of project management expertise as one of the reasons for project failure. As noted earlier, some evidence indicates that CIOB/ITPROD has contributed to strengthening project management capacity. There is, however, some indication that project management expertise may be lacking (particularly for horizontal projects):

  • Most of the 27 senior managers representing Sphere 3 projects who were interviewed believed that CIOB/ITPROD is needed to help sustain their project management capacity.
  • 9 out of 10 case study projects did not meet the original completion schedule, and almost none of the closed-out projects examined were completed on time.
  • Although half of interviewees noted that their departments had project management capacity, they felt that it was not adequate for managing the challenges of complex, horizontal initiatives. Turnover among project team members and maintaining adequate project capacity over the course of multi-year assignments also remain challenges. Close to half of senior management interviewees also suggested that efforts be made to increase project managers' horizontal management skills. They noted that it can be a challenge to attract and retain certified and experienced project managers. It was suggested that strategies be developed to sustain ongoing project management capacity in departments. More than a quarter of respondents stated that they use consultants to offset the shortage of project management talent in their organizations.
  • Although the evaluators did not probe for views related to project management vision, a quarter of senior managers expressed the need for a vision and framework for managing joint projects. A few interviewees indicated that the GC has only a small cadre of project managers who have the necessary capabilities to manage high-risk, complex IT-enabled projects, and that a project management culture is not yet universal in the GC.
  • Although the evaluation does not provide a recommendation on project management capacity, in the opinion of the evaluators it would be beneficial for CIOB/ITPROD, the Office of the Comptroller General and OCHRO to discuss the issue, with a view to determining the need for an overall strategy.

4.2.2 Efficiency and Economy

Evaluation Questions: Is CIOB/ITPROD undertaking its activities and delivering products effectively at the lowest possible cost? How could the cost-effectiveness of program activities be improved?

Findings

There is evidence to suggest that CIOB/ITPROD has functioned efficiently and economically. A lack of performance measurement data and the extensive resource reductions make it difficult to draw a clear conclusion on this issue.

Based on interviews and the review of available documentation, the evaluation found that CIOB/ITPROD was able to administer its activities (although in reduced form) despite resourcing challenges. However, some DGs indicated that, in recent years, CIOB/ITPROD has become more focused on informing Treasury Board ministers, and less so on supporting departments in managing projects.

CIOB/ITPROD interviewees indicated that their ability to carry out their activities effectively is limited by a lack of resources, including the lack of a large enough core of permanent staff. Staffing actions were often delayed, and there was a high level of turnover. A high turnover affects corporate memory, overall capacity and efficiency. In addition, overall demands and the number of projects requiring oversight has been increasing while the CIOB/ITPROD budget has been decreasing.

As part of government-wide exercises, budget cuts took place in the Secretariat for the 2012–13 and 2013–14 fiscal years. CIOB/ITPROD faced a cut of 12 per cent from the salary budget in each year and a cut of 174 per cent in goods and services for 2012–13, because of the requirements of the Deficit Reduction Action Plan. Overall, program documentation indicated that in 2012–13, extensive budgetary cuts required that programs and services be significantly curtailed.

CIOB/ITPROD has received additional pressures from the following:

  • The increase in high-risk, complex IT-enabled projects requiring oversight;
  • All Treasury Board-approved IT-enabled projects must be monitored, which is likely to amount to a significant increase in the number of projects;
  • Departments asking for support (e.g., feedback on dashboards); and
  • The upcoming requirement to renew policies.

Given those pressures, it is not clear whether CIOB/ITPROD will be able to undertake its activities at any lower cost without eliminating more activities.

In discussing the current oversight process, some interviewees (DGs, ADMs, and CIOB/ITPROD staff) indicated a need for reduced reporting frequency, from monthly to every two months or to quarterly, to be more in line with the rate of change in project status. In light of the CIOB/ITPROD resource reductions, this may be an area where efficiencies could be gained.

4.2.3 Oversight Model

Evaluation Questions: Is the CIOB/ITPROD model of oversight and enablement still appropriate? What are the best practices and lessons learned as a result of the CIOB/ITPROD oversight role?

Findings

Canada is considered a leader in project management best practices, and its model parallels other jurisdictions, with the exception of the United States. Nonetheless, some aspects of other models may be worthy of consideration.

Comparison With Models Used in Other Jurisdictions

The evaluation team reviewed the oversight models of Ontario, the United States, Australia, and the United Kingdom. The Canadian model was found to have parallels with some of these.

United States

The US Office of Management and Budget has oversight responsibilities for all IT projects carried out by US departments and agencies. The Office of Management and Budget uses a system known as PortfolioStat. Departments are expected to implement their own oversight programs. PortfolioStat convenes “sessions”–annual face-to-face, evidence-based reviews of an agency’s IT portfolio. These sessions aim to assess the current maturity of each agency’s IT portfolio management process. The IT Dashboard website is one of the key instruments used to oversee federal IT projects. It provides relevant information on the effectiveness of government IT programs and can be searched by anyone for information on both active and discontinued projects.

Ontario

The Ontario government’s oversight structure for IT projects is complex. It includes responsibilities throughout eight Information and Information Technology (I&IT) business clusters that span most provincial ministries and offices, under the umbrella of the I&IT Organization.

Proposed projects whose estimated cost is $10 million or greater require approval at Gate 1, which is provided by the Management Board of Cabinet upon recommendation by the Information Technology Project Approval Committee. Approval is required from the Project Sponsor after Gates 2 through 5. An independent, third-party review is also required.

Australia

The Australian federal government shares responsibility and accountability for IT project oversight with various agencies and departments. Proposals of approximately $30 million or more that include information and communications technology (ICT) costs of at least $10 million, are considered to be high-risk by the Ministry of Finance. These projects are subject to oversight in the form of the ICT Two Pass Review Process. Projects of this size are then subject to assurance reviews in the form of “Gateway Reviews.”

United Kingdom

In the United Kingdom, oversight of ICT spending is the responsibility of the Major Projects Authority, a centralized body tasked with overseeing major projects for the whole of government (whether or not they include IT). There does not appear to be a specific framework for ICT projects, and all major projects are subjected to a similar process. In March 2011, the Cabinet Office announced that all new ICT spending above £5 million would be reviewed by the newly formed Major Projects Authority.

The evaluation team searched for evaluations of the various models, but none was found.

Is the CIOB/ITPROD model still appropriate?

The international comparison literature review found that Canada is considered a leader in best practices in project management. This review also revealed a few promising ideas that could inform a Canadian approach, including the following:

  • Lending of a pool of experts from the centre to departments (Ontario);
  • A shift from project oversight to a portfolio management approach (US);
  • Greater transparency requirements, i.e., sharing independent review findings with the public (US); and
  • Increased emphasis on web-based project reporting, updated on an ongoing basis, with special reviews conducted as needed (US).

Departments suggested retaining the current model but augmenting or shifting the support provided. The following examples were provided:

  • A portfolio management approach, which facilitates government-wide initiatives (In 2011, this approach was identified as a goal in CIOB/ITPROD’s three-year business strategy.);
  • A centre of excellence model (Generally, a centre of expertise could advise departments, and undertake and disseminate research and best practices.);
  • Increased focus on a community of practice model for sharing best practices across the GC;
  • A well-resourced team of experts in IT and project management who use a more hands-on approach across government where necessary;
  • Ensuring that there is a cadre of project managers and training available across the government; and
  • Providing an effective procurement vehicle for external support.

Although Canada is considered a leader in project management best practices, the document review showed that CIOB/ITPROD continues to strive to improve. For example, the program has been moving toward using a portfolio approach to project management. The options presented in this section may assist in the overall consideration of the CIOB/ITPROD function at a higher level.

5. Conclusions

Relevance

  1. CIOB/ITPROD's oversight and enablement program remains relevant. There is a continued need for CIOB/ITPROD to support IT project management capacity, the development of collaborative management approaches and project conception.
  2. CIOB/ITPROD's oversight and enablement program is aligned with Government of Canada (GC) roles, responsibilities and priorities. It is also aligned with the Secretariat's strategic outcome.

Performance: Effectiveness, Efficiency and Economy

  1. Effectiveness
    1. CIOB/ITPROD has achieved its short-term outcomes to a great extent.
      • All lines of evidence demonstrated increased senior management and ministerial awareness of risks and issues, and best practices in the management, governance and oversight of IT-enabled projects.
      • Departments with IT-enabled projects under CIOB/ITPROD's oversight were aware of the tools and guidelines. They also used and applied these tools in projects not subject to CIOB/ITPROD's oversight.
    2. CIOB/ITPROD has progressed in the achievement of its intermediate outcomes.
      1. Senior managers have a greater ability to make timely, evidence-based decisions and project course corrections. The timeliness of CIOB/ITPROD's feedback and its level of involvement, however, did not always meet departments' needs.
      2. Enabling tools, guidance and best practices are being applied to a large extent in the management, governance and oversight of high-risk, complex IT-enabled projects.
      3. Although independent reviews are useful when done well, there is room for improvement in the following areas:
        • The length of time required to contract a reviewer; and
        • The quality of independent reviews.
    3. The long-term outcomes were achieved to some extent.
      1. CIOB/ITPROD has contributed to putting in place the conditions for the success of high-risk, complex IT-enabled projects. However, there is some evidence that the actual achievement of business outcomes is not consistently monitored and evaluated. As a result, the evaluators could not fully assess the extent of contribution to achievement of business outcomes.
      2. Strategies for sound management and governance of IT-enabled projects are in place to some extent. Progress is still needed in the area of performance measurement.
      3. Although there is some indication that project management expertise may be lacking, particularly for horizontal projects, it is not clear to what extent this is the case. A lack of project management expertise is a risk factor for project failure.
  2. Efficiency and Economy

    The CIOB/ITPROD oversight and enablement program appears to function efficiently and economically. However, the lack of performance measurement data and extensive resource reductions make it difficult to draw a clear conclusion on this issue.

  3. Oversight Model

    Canada is considered a leader in project management best practices, and its oversight model and enablement of high-risk, complex IT-enabled projects parallels other jurisdictions, with the exception of the United States. Nonetheless, some aspects of other models may be worthy of consideration.

Recommendations

  1. That CIOB/ITPROD reassess its model of oversight in relation to its resources. It should then develop a revised logic model and performance measurement strategy.
  2. That CIOB/ITPROD, in consultation with the Office of the Comptroller General and other relevant stakeholders, explore options to enhance the ability of federal departments and agencies to monitor and evaluate the business outcomes of their IT-enabled projects. The recommendation does not fall solely within the mandate of CIOB-ITPROD.
  3. That CIOB/ITPROD discuss the independent review program with the appropriate stakeholders, with a view to strengthening it.
  4. That CIOB/ITPROD consider options to streamline the departmental reporting requirements, with a view to enhancing efficiencies and reducing the reporting burden.

Appendix: Documents Reviewed

(Author unknown) (2011). General Overview of TBS [Treasury Board of Canada Secretariat] Role Info for OAG re: Audit of Large IT projects. (Internal document)

Office of the Auditor General of Canada (2011). Status Report of the Auditor General of Canada to the House of Commons: Large Information Technology Projects.

Office of the Auditor General of Canada (2008). Managing Information Technology Investments.

Treasury Board of Canada Secretariat (Date unknown). Annual Monitoring on OAG Recommendations. (Internal document)

Treasury Board of Canada Secretariat (Date unknown). Chief Information Officer Branch Integrated Business Plan 2012-2015. (Internal document)

Treasury Board of Canada Secretariat (Date unknown). Government-wide IT project Monitoring. (Internal document)

Treasury Board of Canada Secretariat (Date unknown). Letter to The Honourable Shawn Murphy, P.C., M.P., in regard to Recommendation 7 of the Seventh Report of the Standing Committee on Public Accounts (39th Parliament, second session) and our commitment to conduct further analysis and report back on an optimal course of action for the implementation of Recommendation 7 by December 31, 2008. (Internal document)

Treasury Board of Canada Secretariat (Date unknown). Letter to Ms. Corinne Charette to endorse the nomination of the TBS CIOB Project Oversight Program team for a Public Service Excellence Award for Innovation. (Internal document)

Treasury Board of Canada Secretariat (Date unknown). Project Governance and Oversight. (Internal document)

Treasury Board of Canada Secretariat (Date unknown). Response to the Seventh Report of the Standing Committee on Public Accounts: Large Information Technology Projects. Analysis and project plan (Internal document)

Treasury Board of Canada Secretariat (2012). ITPROD Resource Allocation – Spreadsheet (As of September 14, 2012). (Internal document)

Treasury Board of Canada Secretariat (2012). IT Project Review and Oversight Division – Strategic Initiatives – Visio Chart. (Internal document)

Treasury Board of Canada Secretariat (2011). CIOB Project Oversight Approach: Ensuring IT-Enabled Project Excellence. (Internal document)

Treasury Board of Canada Secretariat (2011). IT Project Review & Oversight Division (ITPROD) 3-Year Business Strategy. (Internal document)

Treasury Board of Canada Secretariat (2011). Public Service Award of Excellence 2012 – Nomination Form. (Internal document)

Treasury Board of Canada Secretariat (2010). Frequently Asked Questions: Implementation of the Policy on the Management of Projects and the Policy on Investment Planning – Assets and Acquired Services.

Treasury Board of Canada Secretariat (2010). IT Project Review and Oversight Division – Ongoing Commitments – Visio chart. (Internal document)

Treasury Board of Canada Secretariat (2010). IT Project Review and Oversight Division – Strategic Initiatives – Visio Chart. (Internal document)

Treasury Board of Canada Secretariat (2010). Overview of the Treasury Board (TB) Submission Process. (Internal document)

Treasury Board of Canada Secretariat (2009). Opening Remarks by Corinne Charette, Chief Information Officer of the Government of Canada to the Standing Committee on Government Operations and Estimates Follow-up on the Management of Large IT Projects. (Internal document)

Treasury Board of Canada Secretariat (2007). Improving IT Project Performance: Conception, Assessment and Monitoring: An Investigation into the Causes of IT Project Failures in the Government of Canada and a Recommended Course of Action for Improvement. (Internal document)

Treasury Board of Canada Secretariat and Brainhunter (2010). IT-enabled Project Oversight (ITPO) Business Case: An Independent Assessment. (Internal document)

Management Response and Action Plan for the Evaluation of the Oversight and Enablement of High-Risk, Complex IT-Enabled Projects

The Information Technology Project Review and Oversight Division of the Chief Information officer Branch (CIOB/ITPROD) within the Treasury Board of Canada Secretariat welcome the following conclusions from the evaluation study:

CIOB/ITPROD agrees with all the recommendations and proposes the following action plan to address them.

Recommendation 1

That CIOB/ITPROD reassesses its model of oversight in relation to its resources. It should then develop a revised logic model and performance measurement strategy.

Management Response: Agree

Context of the evaluation

The evaluation concluded that there is on ongoing need for CIOB/ITPROD's oversight and enablement program, because this function remains relevant and aligned with Government of Canada (GC) roles, responsibilities and priorities. More importantly, CIOB/ITPROD's work objectives continue to be aligned with the Secretariat's strategic outcome. Participants in this study recognized that the current oversight model is still appropriate and expressed the need for CIOB/ITPROD to increase or shift its support to provide its services to a wider pool of departments and agencies.

Below are planned actions to address Recommendation 1.

Management Action Completion Date Office of Primary Interest
1a. Review the CIOB/ITPROD oversight model in relation to its resources. The review will look at the current environment to ensure that the model continues to support IT-enabled projects and will consider the approaches used in other jurisdictions. By March 2015 CIOB/ITPROD
1b. Through consultations and engagement with appropriate stakeholders, develop a performance measurement strategy for the revised logic model that will be implemented. By March 2015 CIOB/ITPROD

Recommendation 2

That CIOB/ITPROD, in consultation with the Office of the Comptroller General and other relevant stakeholders, explore options to enhance the ability of federal departments and agencies to monitor and evaluate the business outcomes of their IT-enabled projects. This recommendation does not fall solely within the mandate of CIOB-ITPROD.

Management Response: Agree

Context of the evaluation

Much of CIOB/ITPROD's work supports the effective implementation of two Treasury Board policies: the Policy on the Management of Projects and the Policy on Investment Planning–Assets and Acquired Services. In that context, there is no single policy centre to address this recommendation.

The evaluation looked at CIOB/ITPROD's long-term outcomes and noted that it has contributed to put in place conditions for the success of high-risk, complex IT-enabled projects. It also noted that departments did not appear to uniformly collect performance measurement information that could have demonstrated that strong and active governance and oversight may contribute to achieving intended business outcomes of IT-enabled projects. CIOB/ITPROD recognizes the importance in assisting departments and agencies to monitor and evaluate the business outcomes (i.e., benefits realization) in the future.

Below are planned actions to address Recommendation 2.

Management Action Completion Date Office of Primary Interest
2a. CIOB/ITPROD will participate in and contribute to the initiative led by the Office of the Comptroller General (OCG) to review existing project management practices, with the view to determining an approach to improve the ability of federal departments and agencies to report on business outcomes. By March 2015 CIOB/ITPROD
2b. Through consultations and engagement with appropriate stakeholders and in parallel with the OCG led initiative, CIOB/ITPROD will review its guidance documents relating to IT-enabled projects and will update them as required. By September 2015 CIOB/ITPROD

Recommendation 3

That CIOB/ITPROD discuss the independent review program with the appropriate stakeholders, with a view to strengthening it.

Management Response: Agree

Context of the evaluation

In response to the 2006 audit by the Office of the Auditor General, CIOB/ITPROD developed the Independent Review Program for GC IT-enabled projects in 2010.

Much of CIOB/ITPROD's work supports the effective implementation of two TB policies: the Policy on the Management of Projects and the Policy on Investment Planning–Assets and Acquired Services. In that context, departments and agencies are accountable for their projects, the independent reviews and associated procurement processes. CIOB/ITPROD enables them with guidance, tools and recommendations. Whether these recommendations are adopted, or whether other leading practices are adapted, remains a departmental decision.

Below are planned actions to address Recommendation 3.

Management Action Completion Date Office of Primary Interest
3a. To strengthen the Independent Review Program, CIOB/ITPROD will work with Public Works and Government Services Canada to  establish a new procurement approach for departments and agencies to use when they need to acquire independent reviewer services. This will include a guide to improve the quality and consistency of the procurement process for acquiring such services. By June 2015 CIOB/ITPROD
3b. CIOB/ITPROD will gather feedback one year after the implementation of the new procurement approach to evaluate the effectiveness of the Independent Review Program, including the procurement process and the associated guide for acquiring independent reviewer services. One year after implementation of new procurement approach CIOB/ITPROD

Recommendation 4

That CIOB/ITPROD consider options to streamline the departmental reporting requirements, with a view to enhancing efficiencies and reducing the reporting burden.

Management Response: Agree

Context of the evaluation

The evaluation study concluded that departments with IT-Enabled projects under oversight were aware to a great extent of the CIOB/ITPROD guidance documents and tools. In addition, it was found that CIOB/ITPROD tools were used and applied on projects not subject to oversight.

Below is an action planned to address Recommendation 4.

Management Action Completion Date Office of Primary Interest
In consultation with appropriate stakeholders, CIOB/ITPROD will review reporting requirements with a view to minimize reporting burden. CIOB/ITPROD will look for opportunities to improve efficiencies. By December 2015 CIOB/ITPROD
Date modified: