Treasury Board of Canada Secretariat
Symbol of the Government of Canada

ARCHIVED - Audit of the development of the Expenditure Management Information System (EMIS) - Final Report - Management Response - November 25, 2005


Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.


1. Executive Summary

The successful completion of major projects is critical given today's environment of rapid change and limited budgets. It is essential that appropriate project management practices be utilized by Treasury Board Secretariat to ensure the success of their major projects. This audit was requested by the TBS Internal Audit and Evaluation Committee to provide Secretariat management an independent audit of the TBS Expenditure Management Information System (EMIS) Project (Phase I) and to identify areas for improvement. 

EMIS Project

The successful completion of EMIS is critical to support the priorities of TBS and the Government as evidenced by the Budget 2003 announcement for strengthening the Government's Expenditure Management System, reallocation of resources to new priorities, cyclical departmental and horizontal program reviews and linking planning and results management. As of March 2004, EMIS' objectives included providing the Government's Budget Office and Management Board with the information and analytical tools needed to deliver on its stewardship functions. The overall objective is to create an environment that supports the integration of analysis of information for effective expenditure management decisions.

Audit Objectives and Scope

The objective of the internal audit of the EMIS Project (Phase I) is to identify and provide practical feedback and advice regarding significant project management issues. This is the second audit of the EMIS Project; the first was completed in May 2003.  The EMIS project was selected as it represents a highly strategic project for the Government of Canada and in March 2004 an expanded scope was approved. Although this is not a formal follow-up audit, the audit approach was designed to take the findings of the previous audit into consideration.

The scope of this audit primarily focused on project management processes supporting EMIS to deliver a high quality system and new business processes on time and on budget. Treasury Board's Project Management Policy also provided the framework for evaluation of effective project management practices.  

During the course of our audit we received full cooperation and support from the EMIS project team, both at the senior management level and project team level. Although the findings presented in this report are based upon observations made during the course of our audit, it should be noted that management started taking action on many of our findings as they were identified. This reflects the positive attitude towards improving project management of EMIS on an on-going basis.

Observations and Recommendations

The following observations were made during the course of our audit. Detailed findings and recommendations are included in the body of this report.

Absence of Clear Governance and Strong Project Management Practices including the following:

  • There is a lack of a clear governance structure for the project, and the project management mechanisms which are required to ensure appropriate resources are staffed on the project have not been implemented.
  • There is no formal scope management process, and scope of the project is broad and vague.
  • Due to the lack of formal project management mechanisms, senior management must rely upon ad hoc reporting to assess the effectiveness of project management and the resulting economy of the project.
  • The project has achieved some accomplishments, however without formal key milestones and regular progress reporting it is unclear how the project has performed against plan.

We recommend a clear governance structure be developed by senior management. Strong project management resources and processes should be developed and deployed throughout the project. This includes a strengthened Project Management Office, formalized project planning, monitoring, risk management and reporting. If these are not implemented on a timely basis, the achievement of EMIS project objectives will be impaired and the future success of the project is doubtful.

Lack of business transformational and organizational change management experience, approach and methods including the following:

  • An effective change management/business transformation approach is lacking. 
  • The role of business process owners in the testing and acceptance of the system is unclear.
  • Neither the business case for the current phase or future phases are updated for new initiatives, scope changes or changes in costs.
  • There is no Memorandum of Understanding in place with PWGSC that specifies the expectations and requirements of the EMIS project.
  • The approach to communication for the project has been ad hoc, resulting in poor reporting to the steering committee and as well as misunderstandings among the working teams.
  • While the EMIS project is dependant on external contractors and PWGSC for its success, there has been little evidence of the communication and coordination which is required to ensure that they are able to make the appropriate contributions.

We recommend an effective and concerted change management/business transformation approach be implemented that includes organizational readiness, communications, training and deployment strategies.

Further, we recommend that a formal communication plan be developed and implemented. This must include a formal approach to follow-up with third parties including other departments and outside contractors.

Lack of a consistent approach to project methodology and security planning including:

  • There is no methodology road map which ties together the various methodologies used on the project.
  • There is no consistent approach across teams for risk assessment and control.
  • Security implications are currently dealt with by a number of individuals, and it is difficult to assess the completeness of security planning and implementation within the applications.

We recommend that security be given a higher priority on the project and that standard project life cycle and business transformation methodology be fully developed and deployed on the project, consistent with the principles of the TBS Enhanced Management Framework for IM/IT, hereafter referred to as the Enhanced Management Framework.

Conclusion

You will note that many of the observations and recommendations read like a textbook on project management. For every project there are key critical success factors. We used these factors as our audit criteria and we found that in most cases these factors were either absent or not completely implemented. This was not just a "paper exercise" where we simply looked for the existence of specific documents and checked a box if we found them and wrote an observation if we didn't. We examined all aspects of the project and the project management processes. 

The lack of a formal, structured approach to managing this project raises concerns for its ultimate success. We are not suggesting that simply producing specific reports or documents will be sufficient to turning around the project. There has to be a sweeping change to the way this project is run and this includes the tone from the top. The ad hoc "Get to September" approach is not sustainable and needs to be replaced. 

The project needs clear accountabilities and authorities; there needs to be formal structures and processes, and the people have to have the right mix of ability and experience. The project needs to be handled in a textbook fashion since it cannot afford to fail.

Management Response

There is no disagreement with the audit findings. We want the EMIS project to stay on track and succeed and we welcome this input on our methods and performance. We have already addressed a number of the audit findings and provide a summary of our review below.

Context:

Since receiving effective project approval, the EMIS project has had difficulty in several areas:

  • Changing executive direction to the project as TBS priorities moved from a straightforward replacement of the five legacy systems to broader, but less precise, goals to support the transformation of the TBS to a budget office role before this latter goal was adequately defined.
  • Receiving clear direction from a comprehensive set of business process owners on requirements, specifications, and getting decisions about business processes to be implemented in the new suite of systems;
  • Moving from conceptual notions of the project scope and goals to hard plans;
  • Staffing the project with suitable personnel;
  • Accommodating imposed end dates to deliver critical pieces of functionality, which resulted in the IT work being undertaken with short timelines. This resulted in:
    • Quality problems with key deliverables;
    • The necessity of compromising certain fundamental original goals, notably integrating the systems (instead of replacing them one for one) and sharing a common data base (to eliminate duplicate data entry and to address out-of-balance situations between the various systems);
    • Inconsistent technical approaches taken by the development teams who were working in parallel, and had to proceed before architecture work was complete and final technology approach decisions made;
    • The fundamental elements of proper project discipline and control, based on bottom-up estimates, plans, milestones, resource allocation, and reporting, which were in large measure in place at the beginning of the project, were compromised and control effectively lost as these disciplines eroded; and
    • For most of this period, the governance process for the project was ineffective, and no single executive had control or authority over most of the business activity within the scope of the project, resulting in difficulty in securing proper business guidance to and participation in the project.

Certain pieces have nonetheless been delivered. These include: (1) among the legacy replacements, the on-line Annual Reference Level Update; (2) a new system within the approved project scope, the Program Activity Architecture; and (3) new systems not within the approved scope, specifically requests by the Expenditure Review Committee. Two parts of another legacy replacement, on-line Main Estimates and Mains Publishing, are nearing completion. However, the new systems lack integration and have inconsistent technical approaches as noted above. 

Because of (1) the lack of precise definition of the end state; (2) the fact that basic architecture and analysis work had to be curtailed to undertake development to meet target dates for Expenditure Management Information (EMI), PAA, ARLU and Mains; and (3) levels and amounts of IT expertise, and lack of experienced IT direction, there is no current approved "road-map" of how the evolution from the current situation to the end state will be achieved. 

Recent Developments

A recent organizational change at TBS now groups business ownership for much of the scope of EMIS under a single executive (Senior Assistant Secretary – Expensiture Management Sector (EMS)) who is also the project sponsor. Executive management has just approved a new, much more focused and lean governance structure. The department engaged external advice to assist in deciding on corrective actions for the project.

Top management is supportive of changes to improve the project's outlook and create an environment favourable to project success.

Overview of the Corrective Actions

  • The key elements include:
  • Implementing a new governance structure to enable the clarification of business ownership and ensure assignment of proper business expertise to the project team, with clear accountabilities, as well as to establish a locus for end-to-end ownership of business processes, and to advise on work priorities.
  • Following the Treasury Board's Enhanced Management Framework (EMF) and where it is insufficient, consider appropriate, cost-effective and adaptable industry best practices as defined in the Project Management Institute's (PMI) Project Management Book of Knowledge (PMBOK).
  • Encouraging business owners across TBS to adopt a structured approach to change management and business transformation using the Business Transformation Enablement Program (BTEP) as developed and led by the Chief Information Officer Branch (TBS CIOB).
  • Stressing with partners and stakeholders the need to incorporate a structured approach for reference models and patterns for describing programs, services, processes, and resources using where applicable the Government of Canada Strategic Reference Models (GSRM) as developed and led by TBS CIOB in their business processes and business outcomes that EMIS might support.
  • Focusing immediate project activity on:
    • Stable operation of those portions already in production
    • Meeting delivery commitment for Main Estimates
  • Suspending all other development work in favour of:
    • re-confirming details of business requirements with business owners
    • architecture work spanning the range of required functionality that will allow the Roadmap to be completed quickly thereafter;
    • continued analysis of Expenditure Status Report requirements as part of a broader effort to full understand the data needs for the entire scope of the project.
  • Restoring proper project control.
  • Addressing organization, staffing, classification, and filling vacant positions, including Project Director and Head of Information Technology.
  • Creating a new project roadmap, (December 2005) and consequent re-planning of the project (likely January to March 2006)
  • Updating the Effective Project Approval, Project Charter and Business Case to reflect the new approach, with revised cost estimates.
  • Addressing the full inventory of other project issues.

In response to directions from the project sponsor, the EMIS project has made project changes and prepared an action plan that speaks to these audit findings as well as other changes, which will be addressed in each of the sections below.