This page has been archived.
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
3. Observations, Findings and Recommendations
Appendix 2: List of Interviewees
The successful completion of major projects is critical given today's environment of rapid change and limited budgets. It is essential that appropriate project management practices be utilized by Treasury Board Secretariat to ensure the success of their major projects. This audit was requested by the TBS Internal Audit and Evaluation Committee to provide Secretariat management an independent audit of the TBS Expenditure Management Information System (EMIS) Project (Phase I) and to identify areas for improvement.
EMIS Project
The successful completion of EMIS is critical to support the priorities of TBS and the Government as evidenced by the Budget 2003 announcement for strengthening the Government's Expenditure Management System, reallocation of resources to new priorities, cyclical departmental and horizontal program reviews and linking planning and results management. As of March 2004, EMIS' objectives included providing the Government's Budget Office and Management Board with the information and analytical tools needed to deliver on its stewardship functions. The overall objective is to create an environment that supports the integration of analysis of information for effective expenditure management decisions.
Audit Objectives and Scope
The objective of the internal audit of the EMIS Project (Phase I) is to identify and provide practical feedback and advice regarding significant project management issues. This is the second audit of the EMIS Project; the first was completed in May 2003. The EMIS project was selected as it represents a highly strategic project for the Government of Canada and in March 2004 an expanded scope was approved. Although this is not a formal follow-up audit, the audit approach was designed to take the findings of the previous audit into consideration.
The scope of this audit primarily focused on project management processes supporting EMIS to deliver a high quality system and new business processes on time and on budget. Treasury Board's Project Management Policy also provided the framework for evaluation of effective project management practices.
During the course of our audit we received full cooperation and support from the EMIS project team, both at the senior management level and project team level. Although the findings presented in this report are based upon observations made during the course of our audit, it should be noted that management started taking action on many of our findings as they were identified. This reflects the positive attitude towards improving project management of EMIS on an on-going basis.
Observations and Recommendations
The following observations were made during the course of our audit. Detailed findings and recommendations are included in the body of this report.
Absence of Clear Governance and Strong Project Management Practices including the following:
We recommend a clear governance structure be developed by senior management. Strong project management resources and processes should be developed and deployed throughout the project. This includes a strengthened Project Management Office, formalized project planning, monitoring, risk management and reporting. If these are not implemented on a timely basis, the achievement of EMIS project objectives will be impaired and the future success of the project is doubtful.
Lack of business transformational and organizational change management experience, approach and methods including the following:
We recommend an effective and concerted change management/business transformation approach be implemented that includes organizational readiness, communications, training and deployment strategies.
Further, we recommend that a formal communication plan be developed and implemented. This must include a formal approach to follow-up with third parties including other departments and outside contractors.
Lack of a consistent approach to project methodology and security planning including:
We recommend that security be given a higher priority on the project and that standard project life cycle and business transformation methodology be fully developed and deployed on the project, consistent with the principles of the TBS Enhanced Management Framework for IM/IT, hereafter referred to as the Enhanced Management Framework.
Conclusion
You will note that many of the observations and recommendations read like a textbook on project management. For every project there are key critical success factors. We used these factors as our audit criteria and we found that in most cases these factors were either absent or not completely implemented. This was not just a "paper exercise" where we simply looked for the existence of specific documents and checked a box if we found them and wrote an observation if we didn't. We examined all aspects of the project and the project management processes.
The lack of a formal, structured approach to managing this project raises concerns for its ultimate success. We are not suggesting that simply producing specific reports or documents will be sufficient to turning around the project. There has to be a sweeping change to the way this project is run and this includes the tone from the top. The ad hoc "Get to September" approach is not sustainable and needs to be replaced.
The project needs clear accountabilities and authorities; there needs to be formal structures and processes, and the people have to have the right mix of ability and experience. The project needs to be handled in a textbook fashion since it cannot afford to fail.
Management Response
There is no disagreement with the audit findings. We want the EMIS project to stay on track and succeed and we welcome this input on our methods and performance. We have already addressed a number of the audit findings and provide a summary of our review below.
Context:
Since receiving effective project approval, the EMIS project has had difficulty in several areas:
Certain pieces have nonetheless been delivered. These include: (1) among the legacy replacements, the on-line Annual Reference Level Update; (2) a new system within the approved project scope, the Program Activity Architecture; and (3) new systems not within the approved scope, specifically requests by the Expenditure Review Committee. Two parts of another legacy replacement, on-line Main Estimates and Mains Publishing, are nearing completion. However, the new systems lack integration and have inconsistent technical approaches as noted above.
Because of (1) the lack of precise definition of the end state; (2) the fact that basic architecture and analysis work had to be curtailed to undertake development to meet target dates for Expenditure Management Information (EMI), PAA, ARLU and Mains; and (3) levels and amounts of IT expertise, and lack of experienced IT direction, there is no current approved "road-map" of how the evolution from the current situation to the end state will be achieved.
Recent Developments
A recent organizational change at TBS now groups business ownership for much of the scope of EMIS under a single executive (Senior Assistant Secretary – Expensiture Management Sector (EMS)) who is also the project sponsor. Executive management has just approved a new, much more focused and lean governance structure. The department engaged external advice to assist in deciding on corrective actions for the project.
Top management is supportive of changes to improve the project's outlook and create an environment favourable to project success.
Overview of the Corrective Actions
In response to directions from the project sponsor, the EMIS project has made project changes and prepared an action plan that speaks to these audit findings as well as other changes, which will be addressed in each of the sections below.
The successful completion of EMIS is critical to support the priorities of TBS and the Government as evidenced by the Budget 2003 announcement for strengthening the Government's Expenditure Management System, reallocation of resources to new priorities, cyclical departmental and horizontal program reviews and linking planning and results management.
As of March 2004, EMIS' objectives included providing the Government's Budget Office and Management Board with the information and analytical tools needed to deliver on its stewardship functions. The overall objective is to create an environment that supports the integration of analysis of information for effective expenditure management decisions.
Originally and as a result of a series of reviews conducted in the late 1990s, a Preliminary Project Proposal was developed in November 2000 which provided for the EMIS project an amount of $16.2 million over five years. In order to meet commitments outlined in the Budget 2003, the scope was expanded and approved in March 2004 for the current EMIS project which will result in total costs of $53.5 million.
The EMIS Project is currently in Phase 1 of the project. The budget for Phase 1 of EMIS for 2004/05 is $12.7 million and for 2005/06 is $6.7 million. The EMIS project team has reported to the Steering Committee during the latter part of fiscal year 2004/05 that its overall budget for the two years is on track.
Accomplishments from the EMIS Project during 2004/05 include establishment of the Program Activity Architecture (PAA) and the development of a tool for data gathering from the departments/agencies to support Annual Resource Level Update (ARLU) reporting. However, there were a number of project deliverables that were not completed successfully including a reliable technical environment and infrastructure to support data captured from departments, automation of the Estimates process and a sustainable project team with decreasing reliance on outside contractors.
The objective of the internal audit of the EMIS Project (Phase I) is to identify and provide practical feedback and advice regarding significant project management issues. This audit is based on the Treasury Board's Project Management Policy, the objective of which is "To achieve effective and economical management of projects with visible and clearly established project leadership."
The policy requires that projects:
This audit makes practical recommendations, which if implemented, will assist the project in meeting Treasury Board policy's broad objective above.
This audit assessed Phase I of the Expenditure Management Information System (EMIS) Project. The objective of the internal audit is to identify and provide practical feedback and advice regarding significant project management issues. The EMIS project was selected as it represents a highly strategic project for the Government of Canada and in March 2004 an expanded scope was approved. The Internal Audit and Evaluation Committee approved the audit on July 28, 2004. The audit considered the elements of project management and how they apply to the project. The focus of the audit was the current phase and status of EMIS, including the planning for the upcoming phases.
Audits of major systems being developed may have three main components: first, to provide an opinion on the efficiency, effectiveness, and economy of project management; second, to assess the extent to which the system being developed provides for adequate audit trails and controls to ensure the integrity of data processed and stored; and third to assess the controls being provided for the management of the system's operation. The scope of this review focused primarily on the first component: project management processes supporting EMIS to deliver a high quality system and new business processes on time and on budget.
The criteria used for this audit are based on generally accepted criteria and cover the following three general areas:
The detailed criteria for these sections and sub-sections are presented in Appendix 1.
Approach
The approach taken to conduct the audit included the following steps:
Based on the above steps, practical recommendations were developed and communicated to the EMIS project leadership.
Methodology
The audit team conducted research of TB policies and guidelines as well as external best practices for systems under development and business transformation projects. Project management criteria were reviewed and approved by the Audit and Evaluation Committee.
The audit team interviewed key members of the project leadership and project management as well as a number of the members of the EMIS Project Steering Committee. Extensive interviews were conducted with EMIS Project team leads, team members, third party vendor and contractors and external stakeholders. A list of interviewees is included in Appendix 2.
Interviews were also conducted with the Secretary, Assistant Secretaries and representatives from the TBS Services Branch and EMIS business directorate (Estimates, ARLU).
A document review of project documentation including TB Submission, project charter, plans and communications, project status reports, and project budgets was conducted.
Being a Systems Under Development audit, preliminary findings were presented to key stakeholders and project leaders as significant issues were identified during the course of the audit.
Observation
There is a lack of a clear governance structure for the project, and the project management mechanisms which are required to ensure appropriate resources are staffed on the project have not been implemented. |
Findings
Authorities
TBS is the sponsoring department for the project and in that role is required to develop an accountability framework for adequate definition and responsible implementation of the project. We have not seen evidence of such an accountability framework, which is reflected in the following weaknesses:
Resources
Under the Treasury Board Project Management Policy, TBS is responsible for adequately resourcing the project with respect to both funding and the quality and experience of resources. In the case of the project manager, the sponsoring organization did not follow a process which ensured that the project manager had the appropriate level of experience in managing projects of this size, scope and complexity. Furthermore, the stakeholders within TBS did not put in place appropriate resources or follow a process to ensure skills and experience representing the business process owner were in-place to develop requirements and test the initial production system during the fall of 2004.
The project manager has not implemented the mechanisms that would allow him to ensure the project team has all of the necessary competencies required by the project. The level of project planning is not sufficient to identify the specific skills required at specific times; for example, the workplan should identify specific deliverables and the skills or resources required to deliver them, for the current phase of the project. Project plans with detailed resource requirements and assigned tasks are generally only in place for the external consultants. Although there is a general perception that key skills are lacking, it is difficult to substantiate given the lack of planning. Lastly, a formal project organization structure for the project, with well-defined roles and responsibilities was not in place during 2004 and experienced difficulty in staffing project positions and relied heavily on contract resources.
Recommendations
A project accountability framework and project organizational structure should be implemented. Those fulfilling key project management roles must have an appropriate level of experience managing similar projects.
Authorities
A project accountability framework should be developed which defines the following:
Resources
Management Response
Key components of the EMIS project scope have been organized under one sector and therefore much of the EMS requirements and business owners under one Assistant Secretary (Project Sponsor). A new governance model distinct from the departmental governance and with clearly defined roles and responsibilities has been established. Consistent with the accountability regime, the Steering Committee advises and supports the Project Sponsor.
The reorganization within TBS has clarified who many of the business process representatives should be; others, outside of the EMS sector have also been identified
The project's organizational structure with defined roles has been created and approved. The Project Charter will be updated to reflect these new accountabilities.
Clear direction has been given to re-establish the role of the Project Management Office reporting to the Project Director. The Project Management Office will be the sole authority for tracking and reporting on project status against plans.
Staffing strategies for vacant key positions are being developed and, to assess any HR skills and competency shortfalls, a project plan will be developed and a gap analysis performed.
Independent third-party Quality Assurance (QA) assessments will be performed at key project milestones, such as prior to acceptance testing of the Mains Estimates application.
Observation
There is no formal scope management process, and scope of the project is broad and vague. |
Findings
Scope as it exists is both broad and vague. There is no formal scope management process, and as a result there has been no tracking of the impacts of these changes on the original project plan, timelines and budget. There is no clear understanding of key milestones or deliverables, and consequently no defined gates or review points.
Recommendations
A scope management system must be implemented and the scope of the project defined and communicated.
Management Response
Given the reorganization, responsibility for project scope now rests with the project sponsor (Senior Assistant Secretary, Expenditure Management Sector).
Decisions have been made about the current EMIS scope and the project will limit development work to core components while a review is underway of business requirements with the business owners.
The manager on the project responsible for business requirements will recommend trade-offs, as necessary, from among the competing interests of stakeholders through the Project Director to the Project Sponsor who has the authority to decide. Business owner representatives will participate in the architecture process and are responsible to ensure the appropriate scope of functionality under consideration.
The project has revised the scope, project charter and risk management documents to reflect current actions.
Observation
Due to the lack of formal project management mechanisms, senior management must rely upon ad hoc reporting to assess the effectiveness of project management and the resulting economy of the project. |
Findings
Although progress has been made reporting risks to the Steering Committee, there is no formal mechanism in place to capture risks at the execution level of the project, rank them and escalate them to the appropriate levels. In addition, there are no associated mitigation or contingency plans. There is no high level scorecard reporting on the size, scope, complexity, risk, visibility or administrative needs of sub-projects or the project as a whole. Lastly, there has been a lack of effective communication between the Project Team and the Steering Committee as well as the Steering Committee and senior management.
Recommendations
A risk management process needs to be implemented which includes identification, prioritization, mitigation, communication and escalation.
Management Response
The project will be re-sequenced with the development of a Roadmap based on the refined scope resulting from discussions with business owners. During this process, business and project risks at all levels of the organization will be factored into the planning activities for the next phase of the project.
The Project Management Office will assist in the development of a risk-based project plan for each initiative and a risk registry which will be the basis for on-going analysis and reporting.
Project risks will be part of the status report based on a new "executive dashboard" to the Steering Committee, as well as to TBS senior management, and will brought to the Project Sponsor for resolution.
Observation
The project has achieved some accomplishments, however without formal key milestones it is unclear how the project has performed against plan. |
Findings
Progress Reporting
Interviews indicated that the Steering Committee feels that progress reporting has not been at the appropriate level of detail, and both interviews and documentation review have not been able to identify mechanisms that would ensure the content of any such progress reports is based upon a timely and structured roll-up of performance at the delivery level. For example, the Project Charter and workplan identified 69 deliverables under five major workstream during three phases of the EMIS project. However, it is evident that structured budgeting and reporting was not developed within the project team to effectively monitor against these planned objectives and deliverables. In addition, there were only two Steering Committee meetings conducted between the start of the project March 2004 to September 2004. The Steering Committee has been meeting monthly since October 2004; however, there is dissatisfaction with reporting on project performance.
Financial/Budget Reporting
It is difficult to determine where budgets are off-track as there is limited financial/budget reporting and no cost to complete estimates. Budgets are generally held at a high level, not decomposed to major deliverables, activities or sub-activities and all reporting to Steering Committee to date show that deliverables are on budget. Budget authority held for EMIS the Project, and EMIS the Directorate, is blurred and the tracking of financial performance against these authorities is manual and difficult to report on.
Accordingly, there is a high risk that the project will not achieve its expected results at completion without formal tracking and monitoring of results at sufficient detail and without the formality of processes in place for detailed planning, monitoring and risk management.
Recommendations
The project management office should establish key milestones and track progress against those milestones in both practical and financial terms.
Project Tracking
Financial/Budget Tracking
The project management office should establish project control processes over reporting of Actual Spending and Forecast to Completion. As part of the PPMS noted above, incorporate a monthly reporting package including variance analysis, reporting of on track and off track deliverables or work streams and major sub-deliverables to project team members, Steering Committee and TBS senior management.
Management Response
The project management office now has the mandate to re-introduce procedures necessary to assess compliance with project plans. The span of control of the project management office will expand from its present limited scope (development work only) to all project activity. The project management office will educate project staff on planning and estimating, and assist in the development of realistic plans.
Based on the new project sequence (Roadmap) and plan, the Project Charter will be revised; detailed estimates and plans will be developed for a selected portion of the project and appropriate Effective Project Approval authorities sought.
Performance indicators will be established in consultation with business owners and senior management. These will form part of the prototype "Dashboard" to be presented to the Steering Committee in October 2005.
The EMIS project management office will be preparing a monthly executive status report that will include health indicators for each initiative, the status of milestones for each initiative, budget and expenditure information and any risks or issues that require executive attention. The reports will become more comprehensive and useful as the project prepares, and reports against, more detailed plans.
TBS does not have the financial systems to permit the project to collect certain resource data. The project will use proxies, such as person-days, in resourcing, planning and actual reporting to be able to report on financial variance analysis.
Observation
While the EMIS project is dependant on third parties (PWGSC, external contractors) for its success, there has been little evidence of the communication and coordination which is required to ensure that they are able to make the appropriate contributions. |
Findings
EMIS is dependent on third parties (e.g.PWGSC), however, there has been little communication of requirements which would enable these groups to provide the information required.
Although there are contracts in place between EMIS and external contractors, project governance and roles and responsibilities are not clear in this area. More specifically, a number of risks are evident ranging from infrequent performance management between the engagement partners of external contractors and the Project Sponsor to confusion on roles and responsibilities in the area of project management.
Recommendations
There should be a formal approach to communicating requirements and following up on issues with third parties, including PWGSC, other departments and external contractors.
Management Response
The project management office will ensure that all requirements for third parties are formally documented. This will be supplemented by ongoing communications (verbal and written).
The Prime Contractor's ability to contribute will be leveraged by confirming its role on the project management team, and by establishing appropriate regular executive review (metrics-based wherever possible) of both the project and the supplier performance by TBS and Prime Contractor executives.
Process improvements to the way work is assigned to the Prime Contractor will be put into effect to ensure proper estimates, approval of each assignment and a review is carried out.
Consultants and TBS developers are now co-located and knowledge transfer principles and mechanisms will be developed based on Prime Contractor's best practices.
The project management office will re-initiate an issues log and any issues requiring executive attention will be raised immediately and its status reported in the monthly executive status report. An independent test function will report to the Project Director.
Observation
The approach to communication for the project has been ad hoc, resulting in poor reporting to the Steering Committee as well as misunderstandings among the working teams. |
Findings
Communications and reporting have been poor to both the Steering Committee and among the working teams. The approach to communication has been ad hoc and reactive resulting in Steering Committee reporting which is not comprehensive or at the right level of detail and unstructured and inconsistent reporting to TBS senior management. Communications within the project team are ad hoc and not clear or well understood. This has resulted in incomplete or incorrect information being communicated both up and down the chain of command, multiple team members working on tasks without coordination with other team members, and being drawn into new activities in a random manner. Frustration among team members is impacting morale, key project personnel turnover, and creating issues with respect to long term sustainability of the project.
Recommendations
A communications plan must be implemented which is linked to an overall project plan.
Management Response
Effective internal communications on the general concepts and expectations for EMIS have been identified as a necessity.
The project has developed a draft communications plan and strategy that has been shared with TBS Strategic Communications and is being updated to reflect both governance and project management improvements. The communications plan will be linked to the revised project plans.
The communications plan will be presented as well as key messages about the project to EMIS Steering Committee, as well as to the Treasury Board Secretariat's Executive Committee and its Management and Infrastructure Committee.
The skills and experience of large business transformation will be obtained through the staffing of other key positions within the project to assist with communications planning.
Eliminating much of the uncertainty and project ambiguity for the project team (as well as staffing certain key positions), will facilitate team building as well as other changes necessary to operate as a cohesive unit.
Weekly team meetings are scheduled to discuss portfolio, technical and performance management and action items will be recorded and tracked. EMIS employee updates are produced bi-weekly and distributed electronically to the team.
Observation
An effective change management/business transformation approach is lacking. |
Findings
The key mechanism that should be in place to assist with changes across the organization is the involvement of the business process and program area representatives in the testing and rollout of deliverables. However, there have been gaps resulting in system readiness and data input issues.
Recommendations
An effective and concerted change management/business transformation approach must be implemented.
Management Response
We understand the intent of the recommendations but want to stress that EMIS is one part of a larger business transformation at TBS. The Associate Secretary of TBS has been assigned specific responsibility for leading the overall business transformation and change management agenda.
EMIS is an integrating system and, as such, it cannot drive transformational change but can only provide support to that process. Transformational change is a much larger effort underway within all of TBS.
Within the Expenditure Management Sector, a senior position has now been staffed to co-ordinate end-to-end assessment of business processes and of potential for EMIS to contribute to transformational change.
Business process owners and representatives have now been identified and their roles are formalized within the new EMIS governance structure.
Business owners and designated technical authorities have been identified for each business process or activity area within the scope of EMIS. Technical authorities are assigned to the project as their priority assignment. Designated business owners are responsible for specifying requirements, and are responsible for convening advisory groups to ensure appropriate stakeholder input. Business owners have committed to their participation in the development cycle, notably, product validation testing. Final decisions will rest with business owners.
Testing and quality assurance will be done independently and under formal processes and protocols.
Observation
The business case for the current phase or future phases has not been updated for new initiatives, scope changes or changes in costs. |
Findings
The current business case is high level, and has not been updated for new initiatives, scope changes or changes in costs to complete as the underlying project management mechanisms to enable these linkages do not exist. There is no link between events during the current phase and how they impact the business case of future phases. With no formal gate process, and no coordination between the business case and project management processes (because they themselves are not formalized), it is not possible to review the implications of project progress on the business case. Furthermore, there is no process for analyzing how these changes to the current phase impact the business case of future phases.
Recommendation
The business case must be kept up to date to reflect changes in the project, and impacts on future phases must be considered.
Management Response
Business cases will be developed for all new project components and undergo a review and approval process and existing business cases will be updated.
Observation
The role of business process owners in the testing and acceptance of the system is unclear. |
Findings
The EMIS Project has multiple clients within TBS representing the Program Sectors and Expenditure Management Sector. The involvement of business process owners is not well defined and acceptance/sign-off procedures are unclear. There have been differing opinions expressed as to who the clients are for this system. There has not been sufficient time to properly gather requirements, develop, test, train and deploy in the earlier iterations of the deliverables.
Recommendations
Business process owners should formally sign off on deliverables.
Management Response
For each of the EMIS priorities a business owner has been identified and will be responsible for defining the business requirements, acceptance criteria and quality standards. The project will institute several gates in the development cycle for each initiative to determine both EMIS and the client's readiness to proceed to the next phase and if the solution will still meet the business goals. Readiness assessments (lessons learned) will be developed as the project gains experience in delivery solutions.
Observation
There is no Memorandum of Understanding (MOU) in place with PWGSC that specifies the expectations and requirements of the EMIS project. |
Findings
There is no formal MOU in place with PWGSC that specifies the expectations and requirements of the EMIS project. Senior management feels that TBS owns the system and data, and client departments have to use the tools TBS provides for expenditure management. There is considerable lead-time expected by departments to make changes to coding blocks and charts of accounts to meet requirements of TBS and PWGSC financial reporting through the Central Financial Management Reporting System (CFMRS). TBS requirements and timelines have not been developed and communicated on a timely basis. There is a risk that the departmental reporting process may be impaired during FY 2005/06.
Recommendations
An MOU should be put in place with PWGSC that specifies requirements, timeframes and responsibilities.
Management Response
Business process owners - both senior management accountability and the technical process owners - have been identified and the latter will be dedicated to the EMIS team as necessary.
The Senior Information Technology (IT) Executive position will be staffed to lead the IT sections of the project.
A manager responsible for testing and quality assurance will be added and will report to the Project Director. Effective immediately, the testing team will be separated from the development environment and will report to the Project Director.
Agreed implementation plans and strategies will now be included in all EMIS business plans.
The project will assess the best methods to ensure that departments are informed of EMIS initiatives and that they have the capacity to carry out these functions.
Observation
There is no methodology road map that ties together the various methodologies used on the project. Further there is no consistent approach across teams for risk assessment and control. |
Findings
A number of different project methodologies are used on the project, yet there is no roadmap which ties them together to ensure that there are no areas being neglected and that the outputs from one area are compatible with the inputs from another. There is no process in place to ensure that methodology components such as change management and communication are not neglected.
There are no formal tools used for risk assessment and control within the project. Each sub-team within the project has its own approach which does not support a systematic roll-up of this important information.
Recommendations
A standard project life cycle and business transformation methodology needs to be fully developed, training provided and deployed on the project. This must meet the requirements of TBS Management Framework.
The Project Management Office is responsible for methodology documentation and should provide adequate support to the Project Team. The project management office should also ensure policies, procedures and process controls are developed, training is provided and usage is mandatory. Key areas requiring immediate attention to strengthen methods and tools include:
Management Response
A methodologist will be hired, after a review of all project processes, to compile an integrated methodology and to give some structure for future design and architecture work (Phase II).
Gate reviews will be identified in project plans, including the business cases for individual components.
As mentioned previously, testing will be independent and systematic (but an review unit external to the development domain will report directly to the Project Director and not to the project management office).
Quality Assurance (QA) will be factored into project controls. An internal testing and quality assurance function has been established, reporting directly to the Project Head and regular independent quality assurance reviews will be conducted by outside resources.
A review of IT infrastructure to examine cost-effectiveness, plans, establish a budget, and identify cost-reduction possibilities will be undertaken. Also review of the arrangement for systems development services will be carried out to determine the best way to source needed skills as well as development-related software, and methodologies to determine cost effectiveness and opportunities for savings.
Observation
Security implications are currently dealt with by a number of individuals, and it is difficult to assess the completeness of security planning and implementation within the applications. |
Findings
High-level security requirements appear to be understood (e.g. protected environments). However, with no integration of project methodologies, it is difficult to assess the completeness of security planning and implementation within the applications. Security implications are currently dealt with by a number of individuals, but there is no security officer with overall accountability. Security was not well defined in the initial scope definition and was added on during design activities in this phase.
Recommendation
Security must be given a higher priority on the project.
Management Response
The project has implemented a security framework that has been approved by TBS, CSE and PWGSC to ensure tight co-operation of security requirements and testing.
The project has created an Information Systems Security Officer position. This person will operate at the team management level to ensure compliance with the security framework and with the project performance management plan. First results from external reviews of the EMIS project, relative to progress government-wide, have been positive.
You will note that many of the observations and recommendations read like a textbook on project management. For every project there are key critical success factors. We used these factors as our audit criteria and we found that in most cases these factors were either absent or not completely implemented. This was not just a "paper exercise" where we simply looked for the existence of specific documents and checked a box if we found them and wrote an observation if we didn't. We examined all aspects of the project and the project management processes.
The lack of a formal, structured approach to managing this project raises concerns for its ultimate success. We are not suggesting that simply producing specific reports or documents will be sufficient to turning around the project. There has to be a sweeping change to the way this project is run and this includes the tone from the top. The ad hoc "Get to September" approach is not sustainable and needs to be replaced.
The project needs clear accountabilities and authorities; there needs to be formal structures and processes, and the people have to have the right mix of ability and experience. The project needs to be handled in a textbook fashion since it cannot afford to fail.
Management Response
The EMIS team has accepted the TBS Enhanced Management Framework as its textbook for project management. Further, it is encouraging TBS business owners to adopt CIOB's BTEP and GSRM to support their business transformation efforts. Also, consideration of CIOB's work on an outcomes management practice will support and help guide the development of the roadmap and detailed project plans.
EMIS has instituted RUP as its development methodology for EPA Phase I; staff are being trained, and analytical tools acquired, to support these two approaches (EMF and RUP). The Secretary has approved the new EMIS governance model, which clarifies roles and accountabilities.
An overall action plan for planned project changes has been developed, with experienced professional support, to help guide the project sponsor and project director in a process of significant refocusing of the project to meet the recommendations set out in this report, and to further the aims of EMIS in delivering support to the Treasury Board's transformational agenda.
Ruth Dantzer, Associate Secretary
Kevin Page, Executive Director / Project Sponsor EMIS
Mary Jane Jackson, former EMIS Project Co-Sponsor
James Oickle, Project Leader EMIS
Mike Bennett, Project Consultant EMIS
Jim Libbey, Executive Director, Strategic Systems Infrastructure Directorate, Office of the Comptroller General
Helen McDonald, Acting Chief Information Officer
Mike Joyce, Assistant Secretary, Expenditure and Management Strategies Sector
Jim Alexander, Executive Director, IT/IM Stewardship, Chief Information Officer Branch
Daphne Meredith, Assistant Secretary, Economic Sector
Susan Cartwright, Assistant Secretary, Government Operations Sector
Dennis Kam, Acting Assistant Deputy Minister, Corporate Services Branch
Steve Silcox, Acting Executive Director, Finance and Administration, Corporate Services Branch
Marilyn MacPherson, Executive Director, Human Resources Division, Corporate Services Branch
Robert Brodeur, Executive Director, Information Management and Technology Directorate, Corporate Services Branch
Neil Henriksen, Procurement and Risk Management
Mike Delorme, EMIS Project Control Consultant
Arvind Srivastava, Team Lead - Data and Information Management Team
James Gervais, Team Lead, EMIS Administration Management
Duncan Bailey, Senior Research Officer
Paul Joly, Team Lead, EMIS Development Team
Ian Tait, Deloitte Consulting Managing Partner
Arman Mirchandani, Deloitte Consulting EMIS Project Lead
Carlo Beaudoin, Team Lead Business Requirements
Mark Guyan, Team Lead EMIS Operations – Network Services
Josée Desjardins, Team Lead EMIS Communications Team
Bruno Bernier, Director, Central and Public Accounts Reporting, Public Works and Government Services Canada
Richard Patenaude, Director, Central Accounting Systems Directorate, Public Works and Government Services Canada