This page has been archived.
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
The successful completion of major projects is critical given today's environment of rapid change and limited budgets. It is essential that appropriate project management practices be utilized by Treasury Board Secretariat to ensure the success of their major projects. This audit was requested to provide management with an independent assessment of the project management function, the processes for dealing with project risks and the quality of the work performed by the project management teams. To achieve this, one of the major projects currently underway at the Secretariat was selected for review. The project selected was the development of the Expenditure Management Information System ("EMIS").
The purpose of our Project Management Audit was to identify and provide practical feedback and advice regarding project management issues. Our audit was designed to evaluate whether EMIS' project management met the policy requirements of Treasury Board's Project Management policy, and follows industry best practices as defined in the Enhanced Framework for the Management of Information Technology Projects ("EMF-IT") and additional sources. The recommendations made in this report are designed to provide guidance for the EMIS project as it progresses and to help other Secretariat projects with their project management practices.
EMIS Project
A business function of the Treasury Board Secretariat is to operate the Expenditure Management System in support of government-wide expenditure management and planning. In response to concerns regarding the Expenditure Management System's current applications, the Treasury Board approved the initial development of EMIS as an integrated, modern system to support all key functions of the Expenditure Management System. The EMIS Project is being managed within the Expenditure Operations and Estimates Directorate of the Comptrollership Branch. A Project Executive Committee and Steering Committee were established to provide stewardship and guidance to the project management team.
Results and Findings
Overall, we found that the EMIS project is addressing all requirements of the policy. We identified, as part of our audit process, several effective project management practices used by the EMIS project management team.
The following areas were identified for improvement.
Additional areas for improvement are discussed in Section 4.2.
Management Response:
The successful completion of major projects is critical given today's environment of rapid change and limited budgets. It is essential that appropriate project management practices be in place at the Treasury Board Secretariat to ensure the success of major projects. This audit was requested to provide Secretariat management with an independent assessment of the project management function, the processes for dealing with project risks and the quality of the work performed by project teams. To achieve this, one of the major projects currently underway at the Secretariat was selected for review. The project selected was the development of the Expenditure Management Information System (EMIS).
EMIS Project Background1
A business function of the Treasury Board Secretariat is to operate the Expenditure Management System in support of Government-wide expenditure management and planning. A portfolio of applications currently supports the Expenditure Management System, including:
The current applications evolved from the early 1980's independently of one another. They were based upon distinct operational needs related to supporting a particular process (i.e. Main Estimates), concerns for data integrity and specific reporting requirements. A review of the current computer applications by the Expenditure Operations and Estimates Directorate (EOED) of the Comptrollership Branch raised a number of concerns. These included the continuing ability of the technologically obsolete computer applications portfolio to adapt to the Secretariat's changing needs, and the impact (on processes and systems) of emerging business drivers such as the Modern Comptrollership, Improved Reporting to Parliament Project, and Management Board initiatives.
In response to these concerns, the Treasury Board approved the initial development of EMIS as an integrated, modern system to support all key functions of the Expenditure Management System. Specifically, the EMIS system was expected to improve the effectiveness of the Expenditure Management System business functions by:
The EMIS Project is managed by EOED. A project Executive Committee and Steering Committee were established to provide stewardship and guidance to the project management team, and include representation from various branches.
The purpose of our Project Management Audit was to identify and provide practical feedback and advice regarding significant project management issues. Our audit was based on Treasury Board's Project Management policy, and the Enhanced Framework for the Management of Information Technology Projects. Our audit was designed to evaluate whether the EMIS project met policy requirements and its overall objective - "to achieve effective and economical management of projects with visible and clearly established project leadership." Our audit also took into consideration industry best practices for Project Management.
Treasury Board's Project Management policy requires that projects:
This audit assessed the Secretariat's project management practices as utilized for the EMIS project. The objective of the audit was to identify and assist management and the project teams in addressing project management risks and exposures to assess the impact they may have on whether the EMIS project is delivered on time and on budget. The audit considered the elements of project management and how they have been applied to the EMIS project. The focus of the audit was on the current status of the EMIS project and planning for future activities.
The audit assessed the following key project elements:
Provided below is a table identifying the criteria developed for this audit and any corresponding findings related to each criterion. Overall, we found that the EMIS project management team is addressing all requirements of the policy. Further, we identified several areas where improvements would enhance the project management. Specific details on the areas for improvement in the project management are identified, and our recommendations are provided in section 4.2. While many of the recommendations are based on the review of the EMIS project, they are applicable for any project that may be undertaken by the Secretariat.
Audit Objectives2 |
Audit Findings |
|||||||
EMIS Project Related |
Secretariat Related |
|||||||
Project Management Organization |
||||||||
|
4.1.1 |
4.2.2
|
||||||
|
4.1.1 |
4.2.9
|
||||||
|
4.1.2 4.1.5 4.2.3 4.2.6 |
4.2.6
|
||||||
Project Scope and Risk Profile |
||||||||
|
4.1.1 4.1.3 4.2.1 |
4.2.8
|
||||||
|
4.1.1 |
N/A3
|
||||||
|
4.2.7 |
N/A
|
||||||
|
4.2.7 |
N/A
|
||||||
Project Management Practices |
||||||||
|
4.1.4 4.2.4 |
4.2.9
|
We identified as part of our audit process several effective project management practices used by the EMIS project management team.
The EMIS project team developed a project charter for the first phase of the project, Project Definition, defining key aspects of the project including:
Description of the project and definition of the project scope for the initial project definition phase;
Roles and responsibilities for the project team, Project Sponsor, Project Leader, Project Manager, Steering Committee and Executive Committee;
A high level project plan describing the phases and schedule for the project;
Cost estimates by phase of the project;
Project control and guidelines for the management of the project based on the Enhanced Framework for the Management of IT Projects; and
Description of project deliverables for the first phase.
The project team effectively managed the relationship with the external contractor and project resources to limit risk to the project.
A fixed price contract was negotiated to limit the Secretariat's exposure to approved changes.
The request for proposal allowed for project gating at key phases of the project if it is decided to stop the project or change contractors.
Various problems were experienced with internal and external resources, the Project Management Office (PMO) addressed them on a timely basis to minimize the impact on the project.
The PMO was required to devote a significant amount of time to address service delivery issues with the external contractor, which detracted the PMO from dealing with routine duties of the project.
The scope of the EMIS project was defined following a business process improvement study of the EMS, which recommended changes to the process and supporting IT systems. The scope was agreed to in the Preliminary Project Approval (PPA) and approved by the Treasury Board. The Project Definition Phase has progressed based on the scope as defined in the PPA. The project team has effectively managed the project to achieve the scope as initially defined in the project charter and the business process improvement study.
The PMO has put in place several tools for evaluating the project's progress against pre-established timelines and budgets and developed acceptance criteria for the deliverables. These were presented at the Steering and Executive committee meetings.
The Treasury Board Secretariat has not undertaken many projects similar to the size and scope of the EMIS project and therefore, does not have much expertise managing projects of this nature. To compensate for this, the Secretariat brought in, through Executive Interchange, external resources to provide the project management expertise required for this project. There was also an external consultant hired to review and provide advice on the organization of the project management office and project management approach. Secretariat staff was included in the PMO to ensure that the Secretariat was able to develop better project management skills internally.
The following recommendations were developed to address the project management weaknesses and vulnerabilities identified during our audit of the EMIS project management function.
Observations Related to the EMIS Project
Observations Rated High Risk
The scope of the EMIS project was defined following a business process improvement study of the Expenditure Management System, which recommended changes to the process and supporting IT systems. The scope was agreed to in the Preliminary Project Approval (PPA) and approved by the Treasury Board. The Project Definition Phase has progressed based on the scope as defined in the PPA and is almost complete. Even though the scope for the project was approved in the PPA, one of the key stakeholder groups did not fully agree on the appropriateness of the project scope. The PMO took steps during the project definition phase to address this by initiating several joint discussions, and utilizing the services of an arbitrator, in an attempt to resolve the difference of opinion. These efforts are ongoing. While the Project Definition Phase has proceeded as planned, there is evidence that this disagreement has impacted the progress of the project.
We recommend that meetings be held with the Executive and Steering Committees to discuss the project status, and to develop action plans to resolve the differences of opinion before the next phase of the project. A member of Senior Management, with overall responsibility for the key stakeholder groups, should become involved to mediate the discussion. For future projects, we recommend that all key scope issues be resolved prior to the approval of the PPA.
Management Response:
A new EMIS Executive Steering Committee, chaired by an Associate Secretary has been established. This group meets weekly and produces minutes and decision logs for each meeting. In addition a Policy and Process Development Team has been established and is now responsible for defining and obtaining concurrence on the scope of the EMIS Project. Project scope will be formally agreed with the EMIS Executive Steering Committee and documented in a revised Project Charter.
Observations Rated Medium Risk
Project governance for the EMIS project is provided through the Executive and Steering Committees. The committees were established to provide strategic and operational guidance to the project management team. Although the committees consist of representatives from several directorates, some individuals involved with the project indicated that they perceived that there was not equal representation in the committee structure, as one of the Executive Committee Co-chairs, and the Steering Committee Chair are members of the EOED management team. The PMO also reports functionally to the Project Sponsor who is part of EOED. Thus, there is a perception that EOED controls or manages all levels of project management governance for the EMIS project. For some of the people involved with the project, this created a perception that the needs of the EOED group are given more weight than other key stakeholders. While there is no evidence that EOED has exercised undue influence over the project, as EMIS represents a multi-functional project, key stakeholders must have confidence that the project structure will fairly represent the interests of all stakeholders.
We recommend that there be one independent chair for the committee that is in a senior position and able to represent all key stakeholder groups involved in the project. For both the Executive and Steering Committees, the committees' mandate should be formally defined and approved by each committee member. As well, the Project Sponsor should be representative of all key stakeholder groups involved in a project.
Management Response:
A revised governance model is now in place for the EMIS Project. A new EMIS Executive Steering Committee consisting of senior representatives of all key stakeholder groups involved in the project, chaired by an Associate Secretary has been established and a new Project Sponsor has been assigned from outside the Secretariat.
A key factor in the success of any project is the availability of operational staff with knowledge of the Secretariat's operating practices and current applications. These resources are needed to develop detailed system requirements, and review and approve deliverables. The PPA did not provide for back-filling operational staff to allow them to be fully dedicated to the EMIS project at key points during the project definition phase. While funds were ultimately allocated to one of the key stakeholder groups to allow them to "back-fill" staff assigned to the team, they were not used because it was decided that the duties performed by these individuals could not be re-assigned. This put pressure on the operational staff to work on the project while performing their other duties. As well, the availability of operational staff was less than anticipated due to other unexpected demands on their time. It is likely that these were contributing factors in the delay of this project phase.
As recommended in the EMF-IT, clients should be fully involved in all phases of projects to ensure that the applications meet their business requirements in the best manner possible and the expected benefits are obtained. Operational staff should be allocated to projects full-time during key phases to act as liaisons between the developers and the operational group, and provide relevant knowledge of Secretariat operations when planning, prioritizing and conducting project activities. Project budgets should include funding for personnel to back-fill the operational positions of individuals who will be involved in the project. Funding to back-fill positions should be allocated to the project to ensure that it is allocated to project activities and tracked as a project related cost. A detailed resource plan and schedule should identify each resource, and their required level of effort and timing of involvement with the project.
Management Response:
A Policy and Process Development Team has been assembled, comprised of senior representatives from all client operational areas and is now responsible for defining and obtaining concurrence on the scope and business requirements for the EMIS Project. This new group is committed full-time to the EMIS Project and project budgets have been established to permit backfilling of the operational positions for these individuals.
The Executive and Steering Committees were created to provide guidance and direction to the project management team in addressing project issues, developing appropriate courses of action, and performing other activities designed to ensure that the project meets its original objectives. During our review, we identified that the Executive and Steering committees did not always meet on a regular basis. Given the key role these committees play in the success of the project, it is important that the committee members be advised of the project status regularly and that they meet to discuss issues on a timely basis. We also noted that meeting minutes were not prepared to record decisions made, and / or action items developed by the committees.
We recommend that a regular meeting schedule be established for the committees to ensure that the committee members receive timely updates on the project status, and can discuss future timeframes and potential issues. In addition, meeting minutes should be maintained to ensure that all decisions and action plans are recorded. Logs of action items arising from each meeting should be maintained and followed up to ensure they are completed.
Management Response:
The EMIS Executive Steering Committee now meets weekly. Meeting minutes as well as decision and action logs are maintained and distributed to the entire project team subsequent to each meeting.
Observations Rate Low Risk
The PMO has put in place several tools to monitor the progress of the project. During the Project Definition Phase, the PMO has evaluated the project's progress against pre-established timelines and budgets and developed acceptance criteria for the deliverables. The PMO has not developed a formal project performance management system (PPMS) for the entire project as defined in the Treasury Board Project Management Policy. A PPMS consists of a formal set of criteria that are monitored on a regular basis to evaluate the success of a project. It is a management tool for monitoring implementation progress and assessing the likelihood that projects will achieve their development objectives. Although the Treasury Board Project Management Policy only requires a PPMS for Major Crown Projects, and therefore, is not required for EMIS, this is a significant project for the Secretariat and may benefit from a PPMS.
We recommend that the PMO consider developing a formal PPMS as part of the EMIS accountability framework for the remaining phases of the project. The PPMS should include performance indicators linked back to the project objectives, and should provide the PMO and Executive and Steering Committees with adequate information to evaluate the progress of the project against those objectives. A Project Performance Report based on the PPMS should be prepared on a monthly basis that summarizes the performance target levels and provides a comparison against previous results. A qualitative analysis should be prepared for each of the project criteria, which discusses the impact of the result, contributing factors, past and expected future trends, and planned mitigation strategies, if applicable.
Management Response:
A formal project performance management system (potentially Earned Value Measurement System or Balanced Scorecard approach) will be considered in the Build Phase.
Portion of paragraph excluded as per Section 69 of the Access to Information Act. As well, the benefits to be achieved through efficiencies or other improvements to the Expenditure Management process were not assessed. Although it is planned that a full cost-benefit analysis will be provided at the completion of the Project Definition Phase, a significant amount of funds have already been spent on developing the functional requirements of the application. A full analysis of the costs and benefits of the project should have been performed prior to the approval of any funds to ensure the viability of the project, not only in its development but also in future maintenance efforts.
As recommended by the EMF-IT, TBS should prepare a full business case for all future IT projects at the inception of the project, including the full cost of the application from initiation through development, implementation and estimated annual cost of operation. The PMO should ensure that the business case for the EMIS project considers all costs and benefits of the system, including post-implementation maintenance costs.
Management Response:
A full formal business case is being prepared for the EMIS Project, illustrating all costs and benefits of the system, including post-implementation maintenance costs to pursue . project approval.
Throughout the project, a number of documents have been used to identify and analyze the impact of various risks on the project. For example, the Project Charter describes a number of significant project risks, and a Risk Register was drafted to capture risks identified during the project. This latter document was not finalized nor used on an ongoing basis to monitor risks and report status to the Steering and Executive committees. Throughout the project, the PMO was addressing the risks by focusing on delivering the project. While the afore-mentioned documents provide a good basis for monitoring the projects risks, neither of them provides a complete and comprehensive analysis of all known risks, including their potential impact on the project timeframe and cost, influencing factors, mitigating actions and contingency plans as described in Appendix C of the Treasury Board Project Management Policy.
We recommend that, for subsequent phases, a formal, comprehensive Risk Register be developed that includes all identified risks for the project. Following the guidelines provided in Appendix C of the Project Management policy, the risks should be prioritized based on their impact on the project, and their likelihood. Formal mitigation and contingency plans should be developed to address each risk. The Risk Register should be monitored regularly to ensure it remains current, and the risk assessments are still valid. At every meeting the Executive and Steering Committees should be provided a presentation on the status of the most pervasive risks, and their impact on the project.
Management Response:
A formal risk assessment will be conducted prior to starting the next phase of the project.
The risk assessment will result in a prioritised list of the risks that have potential to impact the successful delivery of the project by impacting performance, scope, and cost of schedule. In addition the risk assessment will provide an appropriate set of corresponding contingency plans and mitigation strategies for each identified risk. The results of this assessment will form the basis of the risk log for the remainder of the project.
The EMIS Project Office will maintain the log of all risks that have the potential to affect the project schedule. Specific action plans identified through contingency plans and mitigation strategies will be incorporated into the overall Master Project Schedule and will be monitored through the normal project management cycle.
The project risk log will be reviewed regularly to ensure that all current risks are identified and prioritised appropriately.
Observations related to the Treasury Board Secretariat
Risk Rating: High
Although recommended by the EMF-IT, the TBS has not yet defined a target architecture. A target architecture defines an organization's standard technology and infrastructure components, and is used as a benchmark to ensure compatibility between different IT implementations. For example, EMIS is a multi-functional system that will be used by several TBS directorates, and government departments. As a result, EMIS' compatibility with other IT infrastructures and applications must be considered when developing its IT architecture. By not having a pre-defined target architecture, there is a risk that EMIS' architecture may not be compatible with existing or future applications.
We recommend that TBS develop a target architecture that is appropriate for the organization. Compliance to, or compatibility with, the target architecture should be a requirement for all future IT projects, as recommended in the EMF-IT.
Management Response:
The requirement to formally document the present and target technology architecture has been identified in the departmental IM/IT Strategic Plan that is currently being finalized. It is envisaged that compliance to, or compatibility with, the target architecture will be achieved through the proposed strengthened IM/IT governance structure.
The EMIS Project Office is working with the Information Management and Technology Directorate and Team Deloitte to define a target architecture that is appropriate for the EMIS Project and consistent with the strategic direction of the evolving TBS target architecture.
Risk Rating: Low
Treasury Board Project Management policy requires that departments develop departmental specific policies, guidelines and practices to be used by project management teams responsible for managing major departmental projects. The Secretariat has not developed its own departmental project management polices, guidelines and practices.
We would recommend that a project be initiated to develop departmental policies and procedures for project management. The development of these policies and procedures should include all related divisions within the Secretariat. Once completed, the policies and procedures should be made easily accessible to applicable personnel, and accountability should be assigned for maintaining the policies and procedures.
Management Response:
The departmental IM/IT Strategic Plan, currently being finalized, contains an initiative to develop and implement an effective IM/IT program framework and project management processes. Development of departmental policies and procedures for project management, which are appropriate to the scope of projects normally found within the department, will be incorporated into this initiative. It should be noted that EMIS is being treated as a special case since it is by far the largest internal IM/IT project within the Secretariat in years, and is really out of scope with most Secretariat IM/IT projects. More detailed and extensive project management practices and procedures may be used for EMIS.
A formal Project Management Plan (consistent with standard EMF Project Management guidelines) that details the project management practices and procedures used to manage the EMIS Project is being put in place for the EMIS Project. The EMIS Project Office will facilitate agreement within the EMIS Project Team on the management practices and procedures to be used for the project and will be accountable for developing and maintaining the Project Management Plan.
Provided below is a list of Treasury Board Secretariat employees that we interviewed and/or performed testing with, as part of our audit.
Richard Neville, Deputy Comptroller General |
Comptrollership Branch |
David Bickerton, Executive Director / EMIS Project Sponsor |
Expenditure Operations and Estimates |
Dennis Kam, Executive Director |
Finance and Administration |
Mary Jane Jackson, Director |
Expenditure Strategies |
John Keay, Director |
Expenditure Analysis |
Marc Monette, Senior Financial Analyst |
Expenditure Operations |
Don Grey, EMIS Project Leader |
Comptrollership Branch |
Mike Bennett, EMIS Project Manager |
External Contractor |
Niels Henriksen, Project Coordinator |
Expenditure Management Information System |
Paul Joly, Technical Analyst |
Expenditure Management Information System |
John Huntjens, Project Manager Glen Orsak, Partner |
Deloitte Consulting |
__________________________________________
Endnotes
1 Background extracted from the EMIS Project Charter developed by the Project Team.
2 Objectives are based on the Treasury Board Chapter 2-2 - Project Management policy
3 N/A indicates that there were no criteria within this objective that related generally to the Secretariat.