Treasury Board of Canada Secretariat
Symbol of the Government of Canada

ARCHIVED - Audit of the development of the Expenditure Management Information System (EMIS) - Final Report - Management Response - November 25, 2005

Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.





Audit of the development of the Expenditure
Management Information System (EMIS)

Final Report with Management Response

November 25, 2005






Table of Contents

1. Executive Summary

2. Introduction

3. Observations, Findings and Recommendations

4. Conclusion

Appendix 1: Audit Criteria

Appendix 2: List of Interviewees




1. Executive Summary

The successful completion of major projects is critical given today's environment of rapid change and limited budgets. It is essential that appropriate project management practices be utilized by Treasury Board Secretariat to ensure the success of their major projects. This audit was requested by the TBS Internal Audit and Evaluation Committee to provide Secretariat management an independent audit of the TBS Expenditure Management Information System (EMIS) Project (Phase I) and to identify areas for improvement. 

EMIS Project

The successful completion of EMIS is critical to support the priorities of TBS and the Government as evidenced by the Budget 2003 announcement for strengthening the Government's Expenditure Management System, reallocation of resources to new priorities, cyclical departmental and horizontal program reviews and linking planning and results management. As of March 2004, EMIS' objectives included providing the Government's Budget Office and Management Board with the information and analytical tools needed to deliver on its stewardship functions. The overall objective is to create an environment that supports the integration of analysis of information for effective expenditure management decisions.

Audit Objectives and Scope

The objective of the internal audit of the EMIS Project (Phase I) is to identify and provide practical feedback and advice regarding significant project management issues. This is the second audit of the EMIS Project; the first was completed in May 2003.  The EMIS project was selected as it represents a highly strategic project for the Government of Canada and in March 2004 an expanded scope was approved. Although this is not a formal follow-up audit, the audit approach was designed to take the findings of the previous audit into consideration.

The scope of this audit primarily focused on project management processes supporting EMIS to deliver a high quality system and new business processes on time and on budget. Treasury Board's Project Management Policy also provided the framework for evaluation of effective project management practices.  

During the course of our audit we received full cooperation and support from the EMIS project team, both at the senior management level and project team level. Although the findings presented in this report are based upon observations made during the course of our audit, it should be noted that management started taking action on many of our findings as they were identified. This reflects the positive attitude towards improving project management of EMIS on an on-going basis.

Observations and Recommendations

The following observations were made during the course of our audit. Detailed findings and recommendations are included in the body of this report.

Absence of Clear Governance and Strong Project Management Practices including the following:

  • There is a lack of a clear governance structure for the project, and the project management mechanisms which are required to ensure appropriate resources are staffed on the project have not been implemented.
  • There is no formal scope management process, and scope of the project is broad and vague.
  • Due to the lack of formal project management mechanisms, senior management must rely upon ad hoc reporting to assess the effectiveness of project management and the resulting economy of the project.
  • The project has achieved some accomplishments, however without formal key milestones and regular progress reporting it is unclear how the project has performed against plan.

We recommend a clear governance structure be developed by senior management. Strong project management resources and processes should be developed and deployed throughout the project. This includes a strengthened Project Management Office, formalized project planning, monitoring, risk management and reporting. If these are not implemented on a timely basis, the achievement of EMIS project objectives will be impaired and the future success of the project is doubtful.

Lack of business transformational and organizational change management experience, approach and methods including the following:

  • An effective change management/business transformation approach is lacking. 
  • The role of business process owners in the testing and acceptance of the system is unclear.
  • Neither the business case for the current phase or future phases are updated for new initiatives, scope changes or changes in costs.
  • There is no Memorandum of Understanding in place with PWGSC that specifies the expectations and requirements of the EMIS project.
  • The approach to communication for the project has been ad hoc, resulting in poor reporting to the steering committee and as well as misunderstandings among the working teams.
  • While the EMIS project is dependant on external contractors and PWGSC for its success, there has been little evidence of the communication and coordination which is required to ensure that they are able to make the appropriate contributions.

We recommend an effective and concerted change management/business transformation approach be implemented that includes organizational readiness, communications, training and deployment strategies.

Further, we recommend that a formal communication plan be developed and implemented. This must include a formal approach to follow-up with third parties including other departments and outside contractors.

Lack of a consistent approach to project methodology and security planning including:

  • There is no methodology road map which ties together the various methodologies used on the project.
  • There is no consistent approach across teams for risk assessment and control.
  • Security implications are currently dealt with by a number of individuals, and it is difficult to assess the completeness of security planning and implementation within the applications.

We recommend that security be given a higher priority on the project and that standard project life cycle and business transformation methodology be fully developed and deployed on the project, consistent with the principles of the TBS Enhanced Management Framework for IM/IT, hereafter referred to as the Enhanced Management Framework.

Conclusion

You will note that many of the observations and recommendations read like a textbook on project management. For every project there are key critical success factors. We used these factors as our audit criteria and we found that in most cases these factors were either absent or not completely implemented. This was not just a "paper exercise" where we simply looked for the existence of specific documents and checked a box if we found them and wrote an observation if we didn't. We examined all aspects of the project and the project management processes. 

The lack of a formal, structured approach to managing this project raises concerns for its ultimate success. We are not suggesting that simply producing specific reports or documents will be sufficient to turning around the project. There has to be a sweeping change to the way this project is run and this includes the tone from the top. The ad hoc "Get to September" approach is not sustainable and needs to be replaced. 

The project needs clear accountabilities and authorities; there needs to be formal structures and processes, and the people have to have the right mix of ability and experience. The project needs to be handled in a textbook fashion since it cannot afford to fail.

Management Response

There is no disagreement with the audit findings. We want the EMIS project to stay on track and succeed and we welcome this input on our methods and performance. We have already addressed a number of the audit findings and provide a summary of our review below.

Context:

Since receiving effective project approval, the EMIS project has had difficulty in several areas:

  • Changing executive direction to the project as TBS priorities moved from a straightforward replacement of the five legacy systems to broader, but less precise, goals to support the transformation of the TBS to a budget office role before this latter goal was adequately defined.
  • Receiving clear direction from a comprehensive set of business process owners on requirements, specifications, and getting decisions about business processes to be implemented in the new suite of systems;
  • Moving from conceptual notions of the project scope and goals to hard plans;
  • Staffing the project with suitable personnel;
  • Accommodating imposed end dates to deliver critical pieces of functionality, which resulted in the IT work being undertaken with short timelines. This resulted in:
    • Quality problems with key deliverables;
    • The necessity of compromising certain fundamental original goals, notably integrating the systems (instead of replacing them one for one) and sharing a common data base (to eliminate duplicate data entry and to address out-of-balance situations between the various systems);
    • Inconsistent technical approaches taken by the development teams who were working in parallel, and had to proceed before architecture work was complete and final technology approach decisions made;
    • The fundamental elements of proper project discipline and control, based on bottom-up estimates, plans, milestones, resource allocation, and reporting, which were in large measure in place at the beginning of the project, were compromised and control effectively lost as these disciplines eroded; and
    • For most of this period, the governance process for the project was ineffective, and no single executive had control or authority over most of the business activity within the scope of the project, resulting in difficulty in securing proper business guidance to and participation in the project.

Certain pieces have nonetheless been delivered. These include: (1) among the legacy replacements, the on-line Annual Reference Level Update; (2) a new system within the approved project scope, the Program Activity Architecture; and (3) new systems not within the approved scope, specifically requests by the Expenditure Review Committee. Two parts of another legacy replacement, on-line Main Estimates and Mains Publishing, are nearing completion. However, the new systems lack integration and have inconsistent technical approaches as noted above. 

Because of (1) the lack of precise definition of the end state; (2) the fact that basic architecture and analysis work had to be curtailed to undertake development to meet target dates for Expenditure Management Information (EMI), PAA, ARLU and Mains; and (3) levels and amounts of IT expertise, and lack of experienced IT direction, there is no current approved "road-map" of how the evolution from the current situation to the end state will be achieved. 

Recent Developments

A recent organizational change at TBS now groups business ownership for much of the scope of EMIS under a single executive (Senior Assistant Secretary – Expensiture Management Sector (EMS)) who is also the project sponsor. Executive management has just approved a new, much more focused and lean governance structure. The department engaged external advice to assist in deciding on corrective actions for the project.

Top management is supportive of changes to improve the project's outlook and create an environment favourable to project success.

Overview of the Corrective Actions

  • The key elements include:
  • Implementing a new governance structure to enable the clarification of business ownership and ensure assignment of proper business expertise to the project team, with clear accountabilities, as well as to establish a locus for end-to-end ownership of business processes, and to advise on work priorities.
  • Following the Treasury Board's Enhanced Management Framework (EMF) and where it is insufficient, consider appropriate, cost-effective and adaptable industry best practices as defined in the Project Management Institute's (PMI) Project Management Book of Knowledge (PMBOK).
  • Encouraging business owners across TBS to adopt a structured approach to change management and business transformation using the Business Transformation Enablement Program (BTEP) as developed and led by the Chief Information Officer Branch (TBS CIOB).
  • Stressing with partners and stakeholders the need to incorporate a structured approach for reference models and patterns for describing programs, services, processes, and resources using where applicable the Government of Canada Strategic Reference Models (GSRM) as developed and led by TBS CIOB in their business processes and business outcomes that EMIS might support.
  • Focusing immediate project activity on:
    • Stable operation of those portions already in production
    • Meeting delivery commitment for Main Estimates
  • Suspending all other development work in favour of:
    • re-confirming details of business requirements with business owners
    • architecture work spanning the range of required functionality that will allow the Roadmap to be completed quickly thereafter;
    • continued analysis of Expenditure Status Report requirements as part of a broader effort to full understand the data needs for the entire scope of the project.
  • Restoring proper project control.
  • Addressing organization, staffing, classification, and filling vacant positions, including Project Director and Head of Information Technology.
  • Creating a new project roadmap, (December 2005) and consequent re-planning of the project (likely January to March 2006)
  • Updating the Effective Project Approval, Project Charter and Business Case to reflect the new approach, with revised cost estimates.
  • Addressing the full inventory of other project issues.

In response to directions from the project sponsor, the EMIS project has made project changes and prepared an action plan that speaks to these audit findings as well as other changes, which will be addressed in each of the sections below.

 




2. Introduction

2.1 Background

The successful completion of EMIS is critical to support the priorities of TBS and the Government as evidenced by the Budget 2003 announcement for strengthening the Government's Expenditure Management System, reallocation of resources to new priorities, cyclical departmental and horizontal program reviews and linking planning and results management. 

As of March 2004, EMIS' objectives included providing the Government's Budget Office and Management Board with the information and analytical tools needed to deliver on its stewardship functions. The overall objective is to create an environment that supports the integration of analysis of information for effective expenditure management decisions. 

Originally and as a result of a series of reviews conducted in the late 1990s, a Preliminary Project Proposal was developed in November 2000 which provided for the EMIS project an amount of $16.2 million over five years. In order to meet commitments outlined in the Budget 2003, the scope was expanded and approved in March 2004 for the current EMIS project which will result in total costs of $53.5 million.

The EMIS Project is currently in Phase 1 of the project. The budget for Phase 1 of EMIS for 2004/05 is $12.7 million and for 2005/06 is $6.7 million. The EMIS project team has reported to the Steering Committee during the latter part of fiscal year 2004/05 that its overall budget for the two years is on track.

Accomplishments from the EMIS Project during 2004/05 include establishment of the Program Activity Architecture (PAA) and the development of a tool for data gathering from the departments/agencies to support Annual Resource Level Update (ARLU) reporting. However, there were a number of project deliverables that were not completed successfully including a reliable technical environment and infrastructure to support data captured from departments, automation of the Estimates process and a sustainable project team with decreasing reliance on outside contractors.

2.2 Objective

The objective of the internal audit of the EMIS Project (Phase I) is to identify and provide practical feedback and advice regarding significant project management issues. This audit is based on the Treasury Board's Project Management Policy, the objective of which is "To achieve effective and economical management of projects with visible and clearly established project leadership."

The policy requires that projects:

  • have well defined objectives within an accountability framework;
  • have well defined roles and responsibilities;
  • be approved in accordance with project approval requirements as set out in Chapter 2-1;
  • employ sound project management principles;
  • be adequately resourced;
  • have a comprehensive and coordinated definition of the overall scope of the project; and
  • be managed in a manner sensitive to risk, complexity and economy of resources.

This audit makes practical recommendations, which if implemented, will assist the project in meeting Treasury Board policy's broad objective above.

2.3 Scope

This audit assessed Phase I of the Expenditure Management Information System (EMIS) Project. The objective of the internal audit is to identify and provide practical feedback and advice regarding significant project management issues. The EMIS project was selected as it represents a highly strategic project for the Government of Canada and in March 2004 an expanded scope was approved. The Internal Audit and Evaluation Committee approved the audit on July 28, 2004. The audit considered the elements of project management and how they apply to the project. The focus of the audit was the current phase and status of EMIS, including the planning for the upcoming phases.

Audits of major systems being developed may have three main components: first, to provide an opinion on the efficiency, effectiveness, and economy of project management; second, to assess the extent to which the system being developed provides for adequate audit trails and controls to ensure the integrity of data processed and stored; and third to assess the controls being provided for the management of the system's operation. The scope of this review focused primarily on the first component: project management processes supporting EMIS to deliver a high quality system and new business processes on time and on budget.

2.4 Criteria

The criteria used for this audit are based on generally accepted criteria and cover the following three general areas:

  • Project Management
    • Authorities and Resources
    • Project Scope
    • Project Risk, Complexity and Economy
    • Project Monitoring
    • Business Arrangements
    • Communications
  • Business Transformation and Organizational Readiness
    • Multiple Stakeholders and Business Process Change
    • Business Case
    • Accountability of Projects
    • Accountabilities of Multiple Stakeholders
  • Technology
    • Common Technology, Tools and Methodology
    • IT Security Resources for Projects

The detailed criteria for these sections and sub-sections are presented in Appendix 1.

2.5 Approach and Methodology

Approach

The approach taken to conduct the audit included the following steps:

  • Identified and documented the project management activities and processes used for the EMIS project;
  • Identified project management best practices by researching TB policies and guidelines as well as external project management authorities;
  • Evaluated the EMIS project against audit criteria and best practices; and
  • Identified areas for improvement for the EMIS project.

Based on the above steps, practical recommendations were developed and communicated to the EMIS project leadership.

Methodology

The audit team conducted research of TB policies and guidelines as well as external best practices for systems under development and business transformation projects. Project management criteria were reviewed and approved by the Audit and Evaluation Committee.

The audit team interviewed key members of the project leadership and project management as well as a number of the members of the EMIS Project Steering Committee. Extensive interviews were conducted with EMIS Project team leads, team members, third party vendor and contractors and external stakeholders. A list of interviewees is included in Appendix 2.

Interviews were also conducted with the Secretary, Assistant Secretaries and representatives from the TBS Services Branch and EMIS business directorate (Estimates, ARLU).

A document review of project documentation including TB Submission, project charter, plans and communications, project status reports, and project budgets was conducted.

Being a Systems Under Development audit, preliminary findings were presented to key stakeholders and project leaders as significant issues were identified during the course of the audit.

 




3. Observations, Findings and Recommendations

3.1 Project Management

3.1.1 Authorities and Resources

Observation

There is a lack of a clear governance structure for the project, and the project management mechanisms which are required to ensure appropriate resources are staffed on the project have not been implemented.

Findings

Authorities

TBS is the sponsoring department for the project and in that role is required to develop an accountability framework for adequate definition and responsible implementation of the project. We have not seen evidence of such an accountability framework, which is reflected in the following weaknesses:

  • There is a lack of a clear governance structure for the project that is separate and distinct from the governance structure for the department.
  • There is no clearly defined role for the project sponsor, and there is confusion on who the individual project sponsor is.
  • The role of the EMIS Project Steering Committee is unclear:Is it solely an advisory committee, or does it represent the governing body of the project?The committee met infrequently during 2004, further calling its role and relevance into question. 

Resources

Under the Treasury Board Project Management Policy, TBS is responsible for adequately resourcing the project with respect to both funding and the quality and experience of resources. In the case of the project manager, the sponsoring organization did not follow a process which ensured that the project manager had the appropriate level of experience in managing projects of this size, scope and complexity. Furthermore, the stakeholders within TBS did not put in place appropriate resources or follow a process to ensure skills and experience representing the business process owner were in-place to develop requirements and test the initial production system during the fall of 2004.  

The project manager has not implemented the mechanisms that would allow him to ensure the project team has all of the necessary competencies required by the project. The level of project planning is not sufficient to identify the specific skills required at specific times; for example, the workplan should identify specific deliverables and the skills or resources required to deliver them, for the current phase of the project. Project plans with detailed resource requirements and assigned tasks are generally only in place for the external consultants. Although there is a general perception that key skills are lacking, it is difficult to substantiate given the lack of planning. Lastly, a formal project organization structure for the project, with well-defined roles and responsibilities was not in place during 2004 and experienced difficulty in staffing project positions and relied heavily on contract resources.

Recommendations

A project accountability framework and project organizational structure should be implemented. Those fulfilling key project management roles must have an appropriate level of experience managing similar projects.

Authorities

A project accountability framework should be developed which defines the following:

  • A project governance structure separate from that of the department.
  • Roles and responsibilities of the Project Steering Committee, including accountabilities, management reporting and monitoring.
  • The role of the project sponsor and relationships and accountabilities with the Project Leader and Project Manager roles. Once finalized, these items should be included in the project charter.

Resources

  • The resources assigned to key project management roles must have an appropriate level of experience managing projects of similar size, scope and complexity. 
  • Key business process areas within TBS must ensure the Project Organization is well represented with clearly defined roles and responsibilities, and authorities must be clearly aligned with responsibilities.
  • In order to assess any HR, skills and competency shortfalls, a proper project plan must be in place. 
  • A project management office staffed with the appropriate resources and authority should be implemented to assist team members with project planning and enforce project planning standards. 
  • A project organizational structure should be implemented, formalized in an organization chart, which includes the roles and responsibilities of the team from the Steering Committee down to the user representatives. The project organization needs to be well understood and communicated within TBS and supported by Corporate Services Branch.

Management Response

Key components of the EMIS project scope have been organized under one sector and therefore much of the EMS requirements and business owners under one Assistant Secretary (Project Sponsor). A new governance model distinct from the departmental governance and with clearly defined roles and responsibilities has been established. Consistent with the accountability regime, the Steering Committee advises and supports the Project Sponsor.

The reorganization within TBS has clarified who many of the business process representatives should be; others, outside of the EMS sector have also been identified

The project's organizational structure with defined roles has been created and approved. The Project Charter will be updated to reflect these new accountabilities.

Clear direction has been given to re-establish the role of the Project Management Office reporting to the Project Director. The Project Management Office will be the sole authority for tracking and reporting on project status against plans.

Staffing strategies for vacant key positions are being developed and, to assess any HR skills and competency shortfalls, a project plan will be developed and a gap analysis performed.

Independent third-party Quality Assurance (QA) assessments will be performed at key project milestones, such as prior to acceptance testing of the Mains Estimates application.

3.1.2 Project Scope

Observation

There is no formal scope management process, and scope of the project is broad and vague.

Findings

Scope as it exists is both broad and vague. There is no formal scope management process, and as a result there has been no tracking of the impacts of these changes on the original project plan, timelines and budget. There is no clear understanding of key milestones or deliverables, and consequently no defined gates or review points. 

Recommendations

A scope management system must be implemented and the scope of the project defined and communicated.

  • Project scope needs to be well defined and communicated. Scope management should be part of the responsibilities of the Project Management Office, and an integral part of project planning.  
  • A scope management system must be implemented which allows the balancing of competing demands around scope, time, cost and quality, stakeholders with differing needs and expectations, and changing requirements and expectations. This will ensure all key issues around scope are resolved prior to advancing to further stages, milestones or phases. 

Management Response

Given the reorganization, responsibility for project scope now rests with the project sponsor (Senior Assistant Secretary, Expenditure Management Sector).

Decisions have been made about the current EMIS scope and the project will limit development work to core components while a review is underway of business requirements with the business owners.

The manager on the project responsible for business requirements will recommend trade-offs, as necessary, from among the competing interests of stakeholders through the Project Director to the Project Sponsor who has the authority to decide. Business owner representatives will participate in the architecture process and are responsible to ensure the appropriate scope of functionality under consideration.

The project has revised the scope, project charter and risk management documents to reflect current actions.

3.1.3 Project Risk, Complexity and Economy

Observation

Due to the lack of formal project management mechanisms, senior management must rely upon ad hoc reporting to assess the effectiveness of project management and the resulting economy of the project.

Findings

Although progress has been made reporting risks to the Steering Committee, there is no formal mechanism in place to capture risks at the execution level of the project, rank them and escalate them to the appropriate levels. In addition, there are no associated mitigation or contingency plans. There is no high level scorecard reporting on the size, scope, complexity, risk, visibility or administrative needs of sub-projects or the project as a whole. Lastly, there has been a lack of effective communication between the Project Team and the Steering Committee as well as the Steering Committee and senior management.

Recommendations

A risk management process needs to be implemented which includes identification, prioritization, mitigation, communication and escalation.

  • A formal, comprehensive Risk Management process needs to be implemented within the project organization. A key element of the process includes development of a Risk Register that includes all identified risks for the project. Following the guidelines provided in Appendix C of the Project Management Policy, the risks should be prioritized based on their impact on the project, and their likelihood. Formal mitigation and contingency plans should be developed to address each risk. The Risk Register should be monitored regularly to ensure it remains current, and the risk assessments are still valid. 
  • Potential risk issues need to be documented and escalated to senior management on a timely basis. At every meeting the Executive and Steering Committees should be provided a presentation on the status of the most pervasive risks, and their impact on the project and the risk mitigation activities developed to reduce their impact.

Management Response

The project will be re-sequenced with the development of a Roadmap based on the refined scope resulting from discussions with business owners. During this process, business and project risks at all levels of the organization will be factored into the planning activities for the next phase of the project.

The Project Management Office will assist in the development of a risk-based project plan for each initiative and a risk registry which will be the basis for on-going analysis and reporting.

Project risks will be part of the status report based on a new "executive dashboard" to the Steering Committee, as well as to TBS senior management, and will brought to the Project Sponsor for resolution.

3.1.4 Project Monitoring

Observation

The project has achieved some accomplishments, however without formal key milestones it is unclear how the project has performed against plan.

Findings

Progress Reporting

Interviews indicated that the Steering Committee feels that progress reporting has not been at the appropriate level of detail, and both interviews and documentation review have not been able to identify mechanisms that would ensure the content of any such progress reports is based upon a timely and structured roll-up of performance at the delivery level. For example, the Project Charter and workplan identified 69 deliverables under five major workstream during three phases of the EMIS project. However, it is evident that structured budgeting and reporting was not developed within the project team to effectively monitor against these planned objectives and deliverables. In addition, there were only two Steering Committee meetings conducted between the start of the project March 2004 to September 2004. The Steering Committee has been meeting monthly since October 2004; however, there is dissatisfaction with reporting on project performance.

Financial/Budget Reporting

It is difficult to determine where budgets are off-track as there is limited financial/budget reporting and no cost to complete estimates. Budgets are generally held at a high level, not decomposed to major deliverables, activities or sub-activities and all reporting to Steering Committee to date show that deliverables are on budget. Budget authority held for EMIS the Project, and EMIS the Directorate, is blurred and the tracking of financial performance against these authorities is manual and difficult to report on. 

Accordingly, there is a high risk that the project will not achieve its expected results at completion without formal tracking and monitoring of results at sufficient detail and without the formality of processes in place for detailed planning, monitoring and risk management.

Recommendations

The project management office should establish key milestones and track progress against those milestones in both practical and financial terms.

Project Tracking

  • The project management office should take the lead establishing key milestones and subsequently tracking against these milestones. Project checkpoints should be an integral part of this approach. There are many opportunities for project gates on this project:specific deliverables, user sign-off, software releases, etc. The Project needs to implement regular assessments of project gates and scheduled checkpoints to assess the progress and future of the project. 
  • The project management office needs to develop a formal Project Performance Management System (PPMS) as part of the EMIS accountability framework for the remaining phases of the project. The PPMS should include performance indicators linked back to the project objectives, and should provide the project management office and Executive and Steering Committees with adequate information to evaluate the progress of the project against those objectives. 
  • A Project Performance Report based on the PPMS should be prepared on a regular (e.g. monthly) basis that summarizes the performance target levels and provides a comparison against previous results. A qualitative analysis should be prepared for each of the project criteria, which discusses the impact of the results, contributing factors, past and expected future trends, and planned mitigation strategies, if applicable.

Financial/Budget Tracking

The project management office should establish project control processes over reporting of Actual Spending and Forecast to Completion. As part of the PPMS noted above, incorporate a monthly reporting package including variance analysis, reporting of on track and off track deliverables or work streams and major sub-deliverables to project team members, Steering Committee and TBS senior management. 

Management Response

The project management office now has the mandate to re-introduce procedures necessary to assess compliance with project plans. The span of control of the project management office will expand from its present limited scope (development work only) to all project activity. The project management office will educate project staff on planning and estimating, and assist in the development of realistic plans.

Based on the new project sequence (Roadmap) and plan, the Project Charter will be revised; detailed estimates and plans will be developed for a selected portion of the project and appropriate Effective Project Approval authorities sought.

Performance indicators will be established in consultation with business owners and senior management. These will form part of the prototype "Dashboard" to be presented to the Steering Committee in October 2005.

The EMIS project management office will be preparing a monthly executive status report that will include health indicators for each initiative, the status of milestones for each initiative, budget and expenditure information and any risks or issues that require executive attention. The reports will become more comprehensive and useful as the project prepares, and reports against, more detailed plans.

TBS does not have the financial systems to permit the project to collect certain resource data. The project will use proxies, such as person-days, in resourcing, planning and actual reporting to be able to report on financial variance analysis.

3.1.5 Business Arrangements

Observation

While the EMIS project is dependant on third parties (PWGSC, external contractors) for its success, there has been little evidence of the communication and coordination which is required to ensure that they are able to make the appropriate contributions.

Findings

EMIS is dependent on third parties (e.g.PWGSC), however, there has been little communication of requirements which would enable these groups to provide the information required.

Although there are contracts in place between EMIS and external contractors, project governance and roles and responsibilities are not clear in this area. More specifically, a number of risks are evident ranging from infrequent performance management between the engagement partners of external contractors and the Project Sponsor to confusion on roles and responsibilities in the area of project management.

Recommendations

There should be a formal approach to communicating requirements and following up on issues with third parties, including PWGSC, other departments and external contractors.

  • When dealing with third parties there should be a formal approach to communicating requirements and following up on issues. Normally these items would form part of the project plan and be tracked using standard project management approaches. 
  • When dealing with service providers, such as external contractors, service levels around issues management, performance reporting, project planning, project integration and project risk management need to be defined and agreed upon. Specific areas of the external contractors' relationship which require additional focus include:
    • Knowledge transfer,
    • Issues management,
    • Co-location of consultants and TBS developers,
    • Definition of roles and responsibilities for the management of contracts and Requisitions on Contracts (Statements of work), and
    • Escalation procedures and formal communication.

Management Response

The project management office will ensure that all requirements for third parties are formally documented. This will be supplemented by ongoing communications (verbal and written).

The Prime Contractor's ability to contribute will be leveraged by confirming its role on the project management team, and by establishing appropriate regular executive review (metrics-based wherever possible) of both the project and the supplier performance by TBS and Prime Contractor executives.

Process improvements to the way work is assigned to the Prime Contractor will be put into effect to ensure proper estimates, approval of each assignment and a review is carried out.

Consultants and TBS developers are now co-located and knowledge transfer principles and mechanisms will be developed based on Prime Contractor's best practices.

The project management office will re-initiate an issues log and any issues requiring executive attention will be raised immediately and its status reported in the monthly executive status report. An independent test function will report to the Project Director.

3.1.6 Communications

Observation

The approach to communication for the project has been ad hoc, resulting in poor reporting to the Steering Committee as well as misunderstandings among the working teams.

Findings

Communications and reporting have been poor to both the Steering Committee and among the working teams. The approach to communication has been ad hoc and reactive resulting in Steering Committee reporting which is not comprehensive or at the right level of detail and unstructured and inconsistent reporting to TBS senior management. Communications within the project team are ad hoc and not clear or well understood. This has resulted in incomplete or incorrect information being communicated both up and down the chain of command, multiple team members working on tasks without coordination with other team members, and being drawn into new activities in a random manner. Frustration among team members is impacting morale, key project personnel turnover, and creating issues with respect to long term sustainability of the project.

Recommendations

A communications plan must be implemented which is linked to an overall project plan.

  • Proper communication starts with relevant and timely information. The project management mechanisms recommended elsewhere in this report must be implemented to gather the information, and then appropriate communications to both internal and external stakeholders will be possible. 
  • There should be an overall communication plan linked to the project plan's stages and milestones, and external stakeholder needs should be assessed to ensure that the appropriate information is being gathered and disseminated. 
  • There needs to be skilled and experienced staff with large business transformation experience assigned to the project team/communication team. 
  • Within TBS, internal communication activities must be conducted to promote project awareness (regular information helps develop acceptance of project effort).
  • Project Leadership, the project management office and the Communication team need to invest in approaches to strengthen project communications and performance effectiveness, such as coaching on team communication, team building, and the conduct of effective meetings. 

Management Response

Effective internal communications on the general concepts and expectations for EMIS have been identified as a necessity.

The project has developed a draft communications plan and strategy that has been shared with TBS Strategic Communications and is being updated to reflect both governance and project management improvements. The communications plan will be linked to the revised project plans.

The communications plan will be presented as well as key messages about the project to EMIS Steering Committee, as well as to the Treasury Board Secretariat's Executive Committee and its Management and Infrastructure Committee.

The skills and experience of large business transformation will be obtained through the staffing of other key positions within the project to assist with communications planning.

Eliminating much of the uncertainty and project ambiguity for the project team (as well as staffing certain key positions), will facilitate team building as well as other changes necessary to operate as a cohesive unit.

Weekly team meetings are scheduled to discuss portfolio, technical and performance management and action items will be recorded and tracked. EMIS employee updates are produced bi-weekly and distributed electronically to the team. 

3.2 Business Transformation and Organizational Readiness

3.2.1 Multiple Stakeholders and Business Process Change

Observation

An effective change management/business transformation approach is lacking.

Findings

The key mechanism that should be in place to assist with changes across the organization is the involvement of the business process and program area representatives in the testing and rollout of deliverables. However, there have been gaps resulting in system readiness and data input issues.

Recommendations

An effective and concerted change management/business transformation approach must be implemented.

  • The EMIS project, which has a very broad impact on TBS and departments, needs a very effective and concerted change management/business transformation approach. Key components include organizational readiness, communications, training, and deployment strategies. This must be well planned and established with the skills and competency particularly as the project moves from development to testing, implementation and additional iterative rollouts. 
  • The roles, responsibilities and requirements for business process representatives must be formalized. In cases where the appropriate business process representatives are not available to assist in testing, the issue should be raised on a timely basis to the Steering Committee.

Management Response

We understand the intent of the recommendations but want to stress that EMIS is one part of a larger business transformation at TBS. The Associate Secretary of TBS has been assigned specific responsibility for leading the overall business transformation and change management agenda.

EMIS is an integrating system and, as such, it cannot drive transformational change but can only provide support to that process. Transformational change is a much larger effort underway within all of TBS.

Within the Expenditure Management Sector, a senior position has now been staffed to co-ordinate end-to-end assessment of business processes and of potential for EMIS to contribute to transformational change.

Business process owners and representatives have now been identified and their roles are formalized within the new EMIS governance structure.

Business owners and designated technical authorities have been identified for each business process or activity area within the scope of EMIS. Technical authorities are assigned to the project as their priority assignment. Designated business owners are responsible for specifying requirements, and are responsible for convening advisory groups to ensure appropriate stakeholder input. Business owners have committed to their participation in the development cycle, notably, product validation testing. Final decisions will rest with business owners.

Testing and quality assurance will be done independently and under formal processes and protocols.

3.2.2 Business Case

Observation

The business case for the current phase or future phases has not been updated for new initiatives, scope changes or changes in costs.

Findings

The current business case is high level, and has not been updated for new initiatives, scope changes or changes in costs to complete as the underlying project management mechanisms to enable these linkages do not exist. There is no link between events during the current phase and how they impact the business case of future phases. With no formal gate process, and no coordination between the business case and project management processes (because they themselves are not formalized), it is not possible to review the implications of project progress on the business case. Furthermore, there is no process for analyzing how these changes to the current phase impact the business case of future phases.

Recommendation

The business case must be kept up to date to reflect changes in the project, and impacts on future phases must be considered.

  • A formal process should be implemented to link the business case to the project management metrics that influence it. In addition, there should be a mechanism that ties the resulting updates to the business case to the future phases of the project through the use of sensitivity analysis, steering committee evaluation and approval of the updated business case. This will require previous recommendations related to project management mechanisms to be implemented. 

Management Response

Business cases will be developed for all new project components and undergo a review and approval process and existing business cases will be updated.

  • The process now underway, as part of the Action Plan, of determining the scope of the project, developing the Roadmap and subsequently detailed project plans, will lead to the establishment of viable business cases, which can be tracked over time.
  • A performance management framework including an outcomes management approach, and key performance measures will support business cases, the Roadmap, and detailed project plans. Key elements and corrective actions include:
    • Following the Treasury Board's Enhanced Management Framework (EMF) and where it is insufficient, consider appropriate, cost-effective and adaptable industry best practices as defined in the Project Management Institute's (PMI) Project Management Book of Knowledge (PMBOK);
    • Support and encourage the development of business cases aligned with Business Transformation Enablement Program (BTEP); and
    • Adapting the TBS CIOB's Performance Model from BTEP.

3.2.3 Accountability of Projects

Observation

The role of business process owners in the testing and acceptance of the system is unclear.

Findings

The EMIS Project has multiple clients within TBS representing the Program Sectors and Expenditure Management Sector. The involvement of business process owners is not well defined and acceptance/sign-off procedures are unclear. There have been differing opinions expressed as to who the clients are for this system. There has not been sufficient time to properly gather requirements, develop, test, train and deploy in the earlier iterations of the deliverables. 

Recommendations

Business process owners should formally sign off on deliverables.

  • The project needs to clearly establish business process owners, formal acceptance criteria and dedicated representatives from business areas assigned to project team to take accountability for quality acceptance of deliverables. Business management should have ownership of process alignment.
  • As the project moves into implementation phases, IT and business management must jointly assess implementation strategies and ensure readiness checks and balances are in place. Project management must develop detailed implementation plans with tasks, dates, deliverables, resources assigned and costs. Readiness assessment checklists should be developed and pre-implementation activities well understood, monitored and executed.

Management Response

For each of the EMIS priorities a business owner has been identified and will be responsible for defining the business requirements, acceptance criteria and quality standards. The project will institute several gates in the development cycle for each initiative to determine both EMIS and the client's readiness to proceed to the next phase and if the solution will still meet the business goals. Readiness assessments (lessons learned) will be developed as the project gains experience in delivery solutions.

3.2.4 Accountabilities of Multiple Stakeholders

Observation

There is no Memorandum of Understanding (MOU) in place with PWGSC that specifies the expectations and requirements of the EMIS project.

Findings

There is no formal MOU in place with PWGSC that specifies the expectations and requirements of the EMIS project. Senior management feels that TBS owns the system and data, and client departments have to use the tools TBS provides for expenditure management. There is considerable lead-time expected by departments to make changes to coding blocks and charts of accounts to meet requirements of TBS and PWGSC financial reporting through the Central Financial Management Reporting System (CFMRS). TBS requirements and timelines have not been developed and communicated on a timely basis. There is a risk that the departmental reporting process may be impaired during FY 2005/06.

Recommendations

An MOU should be put in place with PWGSC that specifies requirements, timeframes and responsibilities.

  • There needs to be improved definition of external stakeholders and their relationship to the project. Improved requirements definition and communication with the departments need to be acted upon. Risks of failing to meet required changes and interfaces with CFMRS and EMIS need to be assessed. 
  • Departmental readiness assessments should be conducted for iterations of implementation planned for this phase and Effective Project Approval (EPA) Phase 2.

Management Response

Business process owners - both senior management accountability and the technical process owners - have been identified and the latter will be dedicated to the EMIS team as necessary.

The Senior Information Technology (IT) Executive position will be staffed to lead the IT sections of the project.

A manager responsible for testing and quality assurance will be added and will report to the Project Director. Effective immediately, the testing team will be separated from the development environment and will report to the Project Director.

Agreed implementation plans and strategies will now be included in all EMIS business plans.

The project will assess the best methods to ensure that departments are informed of EMIS initiatives and that they have the capacity to carry out these functions.

3.3 Technology

3.3.1 Common Technology, Tools and Methodology

Observation

There is no methodology road map that ties together the various methodologies used on the project. Further there is no consistent approach across teams for risk assessment and control.

Findings

A number of different project methodologies are used on the project, yet there is no roadmap which ties them together to ensure that there are no areas being neglected and that the outputs from one area are compatible with the inputs from another. There is no process in place to ensure that methodology components such as change management and communication are not neglected.

There are no formal tools used for risk assessment and control within the project. Each sub-team within the project has its own approach which does not support a systematic roll-up of this important information.

Recommendations

A standard project life cycle and business transformation methodology needs to be fully developed, training provided and deployed on the project. This must meet the requirements of TBS Management Framework.

The Project Management Office is responsible for methodology documentation and should provide adequate support to the Project Team. The project management office should also ensure policies, procedures and process controls are developed, training is provided and usage is mandatory. Key areas requiring immediate attention to strengthen methods and tools include:

  • The project should create the role of methodologist, responsible for ensuring that the methodologies used are documented and that they are tied together in a cohesive and logical fashion. This should include the development of a methodology overview that address the various leading methodologies being deployed on the project (e.g. Rational Unified Process (RUP) methodology, Modified Zachman (data architecture methodology) and IT Infrastructure Library (IT Service Management).
  • Proper project gate reviews by business process owners need to be implemented.
  • The project management office needs to develop an organizational wide testing methodology where IT and business process owners develop testing requirements. Further, business users are involved in acceptance testing and with IT, provide written sign-off and approval of each testing process.
  • Quality assessment procedures should be conducted at key project milestones and gates (e.g. Quality Assessment of technology platform prior to deployment to client departments throughout the federal government).

Management Response

A methodologist will be hired, after a review of all project processes, to compile an integrated methodology and to give some structure for future design and architecture work (Phase II).

Gate reviews will be identified in project plans, including the business cases for individual components.

As mentioned previously, testing will be independent and systematic (but an review unit external to the development domain will report directly to the Project Director and not to the project management office).

Quality Assurance (QA) will be factored into project controls. An internal testing and quality assurance function has been established, reporting directly to the Project Head and regular independent quality assurance reviews will be conducted by outside resources.

A review of IT infrastructure to examine cost-effectiveness, plans, establish a budget, and identify cost-reduction possibilities will be undertaken. Also review of the arrangement for systems development services will be carried out to determine the best way to source needed skills as well as development-related software, and methodologies to determine cost effectiveness and opportunities for savings. 

3.3.2 IT Security Resources for Projects

Observation

Security implications are currently dealt with by a number of individuals, and it is difficult to assess the completeness of security planning and implementation within the applications.

Findings

High-level security requirements appear to be understood (e.g. protected environments). However, with no integration of project methodologies, it is difficult to assess the completeness of security planning and implementation within the applications. Security implications are currently dealt with by a number of individuals, but there is no security officer with overall accountability. Security was not well defined in the initial scope definition and was added on during design activities in this phase. 

Recommendation

Security must be given a higher priority on the project.

  • An overall methodology roadmap should be developed which highlights security implications and steps. A security officer role should be formalized to take accountability for security.

Management Response

The project has implemented a security framework that has been approved by TBS, CSE and PWGSC to ensure tight co-operation of security requirements and testing.

The project has created an Information Systems Security Officer position. This person will operate at the team management level to ensure compliance with the security framework and with the project performance management plan. First results from external reviews of the EMIS project, relative to progress government-wide, have been positive.

 




4. Conclusion

You will note that many of the observations and recommendations read like a textbook on project management. For every project there are key critical success factors. We used these factors as our audit criteria and we found that in most cases these factors were either absent or not completely implemented. This was not just a "paper exercise" where we simply looked for the existence of specific documents and checked a box if we found them and wrote an observation if we didn't. We examined all aspects of the project and the project management processes. 

The lack of a formal, structured approach to managing this project raises concerns for its ultimate success. We are not suggesting that simply producing specific reports or documents will be sufficient to turning around the project. There has to be a sweeping change to the way this project is run and this includes the tone from the top. The ad hoc "Get to September" approach is not sustainable and needs to be replaced. 

The project needs clear accountabilities and authorities; there needs to be formal structures and processes, and the people have to have the right mix of ability and experience. The project needs to be handled in a textbook fashion since it cannot afford to fail.

Management Response

The EMIS team has accepted the TBS Enhanced Management Framework as its textbook for project management. Further, it is encouraging TBS business owners to adopt CIOB's BTEP and GSRM to support their business transformation efforts. Also, consideration of CIOB's work on an outcomes management practice will support and help guide the development of the roadmap and detailed project plans.

EMIS has instituted RUP as its development methodology for EPA Phase I; staff are being trained, and analytical tools acquired, to support these two approaches (EMF and RUP). The Secretary has approved the new EMIS governance model, which clarifies roles and accountabilities.

An overall action plan for planned project changes has been developed, with experienced professional support, to help guide the project sponsor and project director in a process of significant refocusing of the project to meet the recommendations set out in this report, and to further the aims of EMIS in delivering support to the Treasury Board's transformational agenda.

Appendix 1:Audit Criteria

  • Project Management
    • Authorities and Resources
      • Project Sponsor is responsible for ensuring that the department understands the value and importance of realizing benefits predicted.
      • From project inception through to ongoing maintenance, sponsoring departments should delegate authorities and allocate adequate resources appropriate to the scope, complexity and risk of the project. 
      • Roles and responsibilities are well defined throughout the life cycle of the project.
      • Project should have mechanisms in place to manage the necessary legislative, regulatory and policy requirements.
    • Project Scope
      • Project leaders are accountable for the full definition of the scope and approval of all scope changes for all projects including the wider interests of the government. 
      • A scope change control system outlines the process for changing the project scope, documentation requirements, tracking systems and approval requirements.
      • Projects, particularly long ones, should have scheduled checkpoints or gates when the project is reviewed and management can decide on the future of the project and any appropriate corrective action. 
      • Gates should be chosen to coincide with logical project review points. Critical path and milestone management is in place and is effective for planning and monitoring scope.
    • Project Risk, Complexity and Economy
      • Project leaders should ensure that project managers perform adequate project planning that addresses the size, scope, complexity, risk, visibility and administrative needs of specific projects. 
      • Project leaders should ensure that the proposed project management framework and allocation of project management resources are based and optimized on the complexity and the assessed risk for the individual phases of a project. 
      • Project management should develop a risk management plan that describes how risk identification, qualitative and quantitative analysis, response planning, monitoring, and control will be structured and performed during the project life cycle. 
      • Senior management should have feedback mechanisms in place to assess the effectiveness of the project management.
    • Project Monitoring
      • The project is performing its intended purpose(s):
        • Monitoring should be focused on continuous improvement and meeting both internal and external performance needs.
        • Regular project performance reports (status reports, quality deliverables/ outcomes) should be prepared by project management and communicated with key stakeholders. 
        • The performance review process should include: status reporting, progress reporting, and forecasting.
    • Business Arrangements
      • Dependence on third parties (e.g. PWGSC).
      • Contractual arrangements (e.g. software/consultants) should be monitored against service levels.
    • Communications
      • Identifying the communication needs of stakeholders, and determining a suitable method for meeting those needs is an important factor for project success. The project management team should determine the project's communication requirements in consultation with the project stakeholders:
        • who needs what information,
        • when they will need it,
        • how it will be given to them and by whom (what decisions/ results).
      • Communications within the project team are clear and understood.
  • Business Transformation and Organizational Readiness
    • Multiple Stakeholders and Business Process Change
      • Culture changes across multiple stakeholders (e.g. sharing of information) and business process changes are addressed through organizational readiness, ongoing communication, education and awareness, and deployment strategies (e.g. training).
    • Business Case
      • The business case should be based on the full cost of the system from initiation through development, implementation and estimated annual cost of operation. 
      • It should be reviewed and revalidated at each scheduled gate and whenever there is a significant change to the project or the business function. 
      • If the business-case of a project changes it should be re-approved through the departmental planning and approval process.
    • Accountability of Projects
      • Clients of systems should be fully involved in all phases of projects, including project gates, to ensure that systems meet their business requirements in the best manner possible and the expected benefits are obtained. 
    • Accountabilities of Multiple Stakeholders
      • The joint commitment and level of involvement of other federal government departments or agencies should be documented in an appropriate interdepartmental agreement, Memoranda of Understanding or equivalent documentation signed by a senior official of each department. 
      • All stakeholders should agree on objectives, roles, deliverables and levels of participation, and should be signed by key senior officials. Interdependencies, both within TBS and with outside departments, need to be well defined (e.g. PWGSC CFMRS).
  • Technology
    • Common Technology, Tools and Methodology
      • In conducting IT projects, departments should conform with approved common infrastructure technology and application and data architectures and infrastructure elements provided by the CIOB of TBS and Government Technology and Information Service of PWGSC, unless there are over-riding operational requirements.
      • Adoption of common IT methodologies and tools used across government should be considered for performing risk assessment and control.  
    • IT Security Resources for Projects
      • In planning new programs, services or major upgrades to existing programs or services, the managers responsible should, at the earliest stage of the funding and approval process, determine the IT security requirements for these programs, services or upgrades and include resource requirements in funding requests.

Appendix 2: List of Interviewees

Ruth Dantzer, Associate Secretary

Kevin Page, Executive Director / Project Sponsor EMIS

Mary Jane Jackson, former EMIS Project Co-Sponsor

James Oickle, Project Leader EMIS

Mike Bennett, Project Consultant EMIS

Jim Libbey, Executive Director, Strategic Systems Infrastructure Directorate, Office of the Comptroller General

Helen McDonald, Acting Chief Information Officer

Mike Joyce, Assistant Secretary, Expenditure and Management Strategies Sector

Jim Alexander, Executive Director, IT/IM Stewardship, Chief Information Officer Branch

Daphne Meredith, Assistant Secretary, Economic Sector

Susan Cartwright, Assistant Secretary, Government Operations Sector

Dennis Kam, Acting Assistant Deputy Minister, Corporate Services Branch

Steve Silcox, Acting Executive Director, Finance and Administration, Corporate Services Branch

Marilyn MacPherson, Executive Director, Human Resources Division, Corporate Services Branch

Robert Brodeur, Executive Director, Information Management and Technology Directorate, Corporate Services Branch

Neil Henriksen, Procurement and Risk Management

Mike Delorme, EMIS Project Control Consultant

Arvind Srivastava, Team Lead - Data and Information Management Team

James Gervais, Team Lead, EMIS Administration Management

Duncan Bailey, Senior Research Officer

Paul Joly, Team Lead, EMIS Development Team

Ian Tait, Deloitte Consulting Managing Partner

Arman Mirchandani, Deloitte Consulting EMIS Project Lead

Carlo Beaudoin, Team Lead Business Requirements

Mark Guyan, Team Lead EMIS Operations – Network Services

Josée Desjardins, Team Lead EMIS Communications Team

Bruno Bernier, Director, Central and Public Accounts Reporting, Public Works and Government Services Canada

Richard Patenaude, Director, Central Accounting Systems Directorate, Public Works and Government Services Canada