ARCHIVED - Audit of the development of the Expenditure Management Information System (EMIS) - Final Report - Management Response - November 25, 2005

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

3. Observations, Findings and Recommendations

3.1 Project Management

3.1.1 Authorities and Resources

Observation

Findings

Authorities

TBS is the sponsoring department for the project and in that role is required to develop an accountability framework for adequate definition and responsible implementation of the project. We have not seen evidence of such an accountability framework, which is reflected in the following weaknesses:

• There is a lack of a clear governance structure for the project that is separate and distinct from the governance structure for the department.
• There is no clearly defined role for the project sponsor, and there is confusion on who the individual project sponsor is.
• The role of the EMIS Project Steering Committee is unclear:Is it solely an advisory committee, or does it represent the governing body of the project?The committee met infrequently during 2004, further calling its role and relevance into question. 

Resources

Under the Treasury Board Project Management Policy, TBS is responsible for adequately resourcing the project with respect to both funding and the quality and experience of resources. In the case of the project manager, the sponsoring organization did not follow a process which ensured that the project manager had the appropriate level of experience in managing projects of this size, scope and complexity. Furthermore, the stakeholders within TBS did not put in place appropriate resources or follow a process to ensure skills and experience representing the business process owner were in-place to develop requirements and test the initial production system during the fall of 2004.  

The project manager has not implemented the mechanisms that would allow him to ensure the project team has all of the necessary competencies required by the project. The level of project planning is not sufficient to identify the specific skills required at specific times; for example, the workplan should identify specific deliverables and the skills or resources required to deliver them, for the current phase of the project. Project plans with detailed resource requirements and assigned tasks are generally only in place for the external consultants. Although there is a general perception that key skills are lacking, it is difficult to substantiate given the lack of planning. Lastly, a formal project organization structure for the project, with well-defined roles and responsibilities was not in place during 2004 and experienced difficulty in staffing project positions and relied heavily on contract resources.

Recommendations

A project accountability framework and project organizational structure should be implemented. Those fulfilling key project management roles must have an appropriate level of experience managing similar projects.

Authorities

A project accountability framework should be developed which defines the following:

• A project governance structure separate from that of the department.
• Roles and responsibilities of the Project Steering Committee, including accountabilities, management reporting and monitoring.
• The role of the project sponsor and relationships and accountabilities with the Project Leader and Project Manager roles. Once finalized, these items should be included in the project charter.

Resources

• The resources assigned to key project management roles must have an appropriate level of experience managing projects of similar size, scope and complexity. 
• Key business process areas within TBS must ensure the Project Organization is well represented with clearly defined roles and responsibilities, and authorities must be clearly aligned with responsibilities.
• In order to assess any HR, skills and competency shortfalls, a proper project plan must be in place. 
• A project management office staffed with the appropriate resources and authority should be implemented to assist team members with project planning and enforce project planning standards. 
• A project organizational structure should be implemented, formalized in an organization chart, which includes the roles and responsibilities of the team from the Steering Committee down to the user representatives. The project organization needs to be well understood and communicated within TBS and supported by Corporate Services Branch.

Management Response

Key components of the EMIS project scope have been organized under one sector and therefore much of the EMS requirements and business owners under one Assistant Secretary (Project Sponsor). A new governance model distinct from the departmental governance and with clearly defined roles and responsibilities has been established. Consistent with the accountability regime, the Steering Committee advises and supports the Project Sponsor.

The reorganization within TBS has clarified who many of the business process representatives should be; others, outside of the EMS sector have also been identified

The project's organizational structure with defined roles has been created and approved. The Project Charter will be updated to reflect these new accountabilities.

Clear direction has been given to re-establish the role of the Project Management Office reporting to the Project Director. The Project Management Office will be the sole authority for tracking and reporting on project status against plans.

Staffing strategies for vacant key positions are being developed and, to assess any HR skills and competency shortfalls, a project plan will be developed and a gap analysis performed.

Independent third-party Quality Assurance (QA) assessments will be performed at key project milestones, such as prior to acceptance testing of the Mains Estimates application.

3.1.2 Project Scope

Observation

Findings

Scope as it exists is both broad and vague. There is no formal scope management process, and as a result there has been no tracking of the impacts of these changes on the original project plan, timelines and budget. There is no clear understanding of key milestones or deliverables, and consequently no defined gates or review points. 

Recommendations

A scope management system must be implemented and the scope of the project defined and communicated.

• Project scope needs to be well defined and communicated. Scope management should be part of the responsibilities of the Project Management Office, and an integral part of project planning.  
• A scope management system must be implemented which allows the balancing of competing demands around scope, time, cost and quality, stakeholders with differing needs and expectations, and changing requirements and expectations. This will ensure all key issues around scope are resolved prior to advancing to further stages, milestones or phases. 

Management Response

Given the reorganization, responsibility for project scope now rests with the project sponsor (Senior Assistant Secretary, Expenditure Management Sector).

Decisions have been made about the current EMIS scope and the project will limit development work to core components while a review is underway of business requirements with the business owners.

The manager on the project responsible for business requirements will recommend trade-offs, as necessary, from among the competing interests of stakeholders through the Project Director to the Project Sponsor who has the authority to decide. Business owner representatives will participate in the architecture process and are responsible to ensure the appropriate scope of functionality under consideration.

The project has revised the scope, project charter and risk management documents to reflect current actions.

3.1.3 Project Risk, Complexity and Economy

Observation

Findings

Although progress has been made reporting risks to the Steering Committee, there is no formal mechanism in place to capture risks at the execution level of the project, rank them and escalate them to the appropriate levels. In addition, there are no associated mitigation or contingency plans. There is no high level scorecard reporting on the size, scope, complexity, risk, visibility or administrative needs of sub-projects or the project as a whole. Lastly, there has been a lack of effective communication between the Project Team and the Steering Committee as well as the Steering Committee and senior management.

Recommendations

A risk management process needs to be implemented which includes identification, prioritization, mitigation, communication and escalation.

• A formal, comprehensive Risk Management process needs to be implemented within the project organization. A key element of the process includes development of a Risk Register that includes all identified risks for the project. Following the guidelines provided in Appendix C of the Project Management Policy, the risks should be prioritized based on their impact on the project, and their likelihood. Formal mitigation and contingency plans should be developed to address each risk. The Risk Register should be monitored regularly to ensure it remains current, and the risk assessments are still valid. 
• Potential risk issues need to be documented and escalated to senior management on a timely basis. At every meeting the Executive and Steering Committees should be provided a presentation on the status of the most pervasive risks, and their impact on the project and the risk mitigation activities developed to reduce their impact.

Management Response

The project will be re-sequenced with the development of a Roadmap based on the refined scope resulting from discussions with business owners. During this process, business and project risks at all levels of the organization will be factored into the planning activities for the next phase of the project.

The Project Management Office will assist in the development of a risk-based project plan for each initiative and a risk registry which will be the basis for on-going analysis and reporting.

Project risks will be part of the status report based on a new "executive dashboard" to the Steering Committee, as well as to TBS senior management, and will brought to the Project Sponsor for resolution.

3.1.4 Project Monitoring

Observation

Findings

Progress Reporting

Interviews indicated that the Steering Committee feels that progress reporting has not been at the appropriate level of detail, and both interviews and documentation review have not been able to identify mechanisms that would ensure the content of any such progress reports is based upon a timely and structured roll-up of performance at the delivery level. For example, the Project Charter and workplan identified 69 deliverables under five major workstream during three phases of the EMIS project. However, it is evident that structured budgeting and reporting was not developed within the project team to effectively monitor against these planned objectives and deliverables. In addition, there were only two Steering Committee meetings conducted between the start of the project March 2004 to September 2004. The Steering Committee has been meeting monthly since October 2004; however, there is dissatisfaction with reporting on project performance.

Financial/Budget Reporting

It is difficult to determine where budgets are off-track as there is limited financial/budget reporting and no cost to complete estimates. Budgets are generally held at a high level, not decomposed to major deliverables, activities or sub-activities and all reporting to Steering Committee to date show that deliverables are on budget. Budget authority held for EMIS the Project, and EMIS the Directorate, is blurred and the tracking of financial performance against these authorities is manual and difficult to report on. 

Accordingly, there is a high risk that the project will not achieve its expected results at completion without formal tracking and monitoring of results at sufficient detail and without the formality of processes in place for detailed planning, monitoring and risk management.

Recommendations

The project management office should establish key milestones and track progress against those milestones in both practical and financial terms.

Project Tracking

• The project management office should take the lead establishing key milestones and subsequently tracking against these milestones. Project checkpoints should be an integral part of this approach. There are many opportunities for project gates on this project:specific deliverables, user sign-off, software releases, etc. The Project needs to implement regular assessments of project gates and scheduled checkpoints to assess the progress and future of the project. 
• The project management office needs to develop a formal Project Performance Management System (PPMS) as part of the EMIS accountability framework for the remaining phases of the project. The PPMS should include performance indicators linked back to the project objectives, and should provide the project management office and Executive and Steering Committees with adequate information to evaluate the progress of the project against those objectives. 
• A Project Performance Report based on the PPMS should be prepared on a regular (e.g. monthly) basis that summarizes the performance target levels and provides a comparison against previous results. A qualitative analysis should be prepared for each of the project criteria, which discusses the impact of the results, contributing factors, past and expected future trends, and planned mitigation strategies, if applicable.

Financial/Budget Tracking

The project management office should establish project control processes over reporting of Actual Spending and Forecast to Completion. As part of the PPMS noted above, incorporate a monthly reporting package including variance analysis, reporting of on track and off track deliverables or work streams and major sub-deliverables to project team members, Steering Committee and TBS senior management. 

Management Response

The project management office now has the mandate to re-introduce procedures necessary to assess compliance with project plans. The span of control of the project management office will expand from its present limited scope (development work only) to all project activity. The project management office will educate project staff on planning and estimating, and assist in the development of realistic plans.

Based on the new project sequence (Roadmap) and plan, the Project Charter will be revised; detailed estimates and plans will be developed for a selected portion of the project and appropriate Effective Project Approval authorities sought.

Performance indicators will be established in consultation with business owners and senior management. These will form part of the prototype "Dashboard" to be presented to the Steering Committee in October 2005.

The EMIS project management office will be preparing a monthly executive status report that will include health indicators for each initiative, the status of milestones for each initiative, budget and expenditure information and any risks or issues that require executive attention. The reports will become more comprehensive and useful as the project prepares, and reports against, more detailed plans.

TBS does not have the financial systems to permit the project to collect certain resource data. The project will use proxies, such as person-days, in resourcing, planning and actual reporting to be able to report on financial variance analysis.

3.1.5 Business Arrangements

Observation

Findings

EMIS is dependent on third parties (e.g.PWGSC), however, there has been little communication of requirements which would enable these groups to provide the information required.

Although there are contracts in place between EMIS and external contractors, project governance and roles and responsibilities are not clear in this area. More specifically, a number of risks are evident ranging from infrequent performance management between the engagement partners of external contractors and the Project Sponsor to confusion on roles and responsibilities in the area of project management.

Recommendations

There should be a formal approach to communicating requirements and following up on issues with third parties, including PWGSC, other departments and external contractors.

• When dealing with third parties there should be a formal approach to communicating requirements and following up on issues. Normally these items would form part of the project plan and be tracked using standard project management approaches. 
• When dealing with service providers, such as external contractors, service levels around issues management, performance reporting, project planning, project integration and project risk management need to be defined and agreed upon. Specific areas of the external contractors' relationship which require additional focus include:
  • Knowledge transfer,
  • Issues management,
  • Co-location of consultants and TBS developers,
  • Definition of roles and responsibilities for the management of contracts and Requisitions on Contracts (Statements of work), and
  • Escalation procedures and formal communication.

Management Response

The project management office will ensure that all requirements for third parties are formally documented. This will be supplemented by ongoing communications (verbal and written).

The Prime Contractor's ability to contribute will be leveraged by confirming its role on the project management team, and by establishing appropriate regular executive review (metrics-based wherever possible) of both the project and the supplier performance by TBS and Prime Contractor executives.

Process improvements to the way work is assigned to the Prime Contractor will be put into effect to ensure proper estimates, approval of each assignment and a review is carried out.

Consultants and TBS developers are now co-located and knowledge transfer principles and mechanisms will be developed based on Prime Contractor's best practices.

The project management office will re-initiate an issues log and any issues requiring executive attention will be raised immediately and its status reported in the monthly executive status report. An independent test function will report to the Project Director.

3.1.6 Communications

Observation

Findings

Communications and reporting have been poor to both the Steering Committee and among the working teams. The approach to communication has been ad hoc and reactive resulting in Steering Committee reporting which is not comprehensive or at the right level of detail and unstructured and inconsistent reporting to TBS senior management. Communications within the project team are ad hoc and not clear or well understood. This has resulted in incomplete or incorrect information being communicated both up and down the chain of command, multiple team members working on tasks without coordination with other team members, and being drawn into new activities in a random manner. Frustration among team members is impacting morale, key project personnel turnover, and creating issues with respect to long term sustainability of the project.

Recommendations

A communications plan must be implemented which is linked to an overall project plan.

• Proper communication starts with relevant and timely information. The project management mechanisms recommended elsewhere in this report must be implemented to gather the information, and then appropriate communications to both internal and external stakeholders will be possible. 
• There should be an overall communication plan linked to the project plan's stages and milestones, and external stakeholder needs should be assessed to ensure that the appropriate information is being gathered and disseminated. 
• There needs to be skilled and experienced staff with large business transformation experience assigned to the project team/communication team. 
• Within TBS, internal communication activities must be conducted to promote project awareness (regular information helps develop acceptance of project effort).
• Project Leadership, the project management office and the Communication team need to invest in approaches to strengthen project communications and performance effectiveness, such as coaching on team communication, team building, and the conduct of effective meetings. 

Management Response

Effective internal communications on the general concepts and expectations for EMIS have been identified as a necessity.

The project has developed a draft communications plan and strategy that has been shared with TBS Strategic Communications and is being updated to reflect both governance and project management improvements. The communications plan will be linked to the revised project plans.

The communications plan will be presented as well as key messages about the project to EMIS Steering Committee, as well as to the Treasury Board Secretariat's Executive Committee and its Management and Infrastructure Committee.

The skills and experience of large business transformation will be obtained through the staffing of other key positions within the project to assist with communications planning.

Eliminating much of the uncertainty and project ambiguity for the project team (as well as staffing certain key positions), will facilitate team building as well as other changes necessary to operate as a cohesive unit.

Weekly team meetings are scheduled to discuss portfolio, technical and performance management and action items will be recorded and tracked. EMIS employee updates are produced bi-weekly and distributed electronically to the team. 

3.2 Business Transformation and Organizational Readiness

3.2.1 Multiple Stakeholders and Business Process Change

Observation

Findings

The key mechanism that should be in place to assist with changes across the organization is the involvement of the business process and program area representatives in the testing and rollout of deliverables. However, there have been gaps resulting in system readiness and data input issues.

Recommendations

An effective and concerted change management/business transformation approach must be implemented.

• The EMIS project, which has a very broad impact on TBS and departments, needs a very effective and concerted change management/business transformation approach. Key components include organizational readiness, communications, training, and deployment strategies. This must be well planned and established with the skills and competency particularly as the project moves from development to testing, implementation and additional iterative rollouts. 
• The roles, responsibilities and requirements for business process representatives must be formalized. In cases where the appropriate business process representatives are not available to assist in testing, the issue should be raised on a timely basis to the Steering Committee.

Management Response

We understand the intent of the recommendations but want to stress that EMIS is one part of a larger business transformation at TBS. The Associate Secretary of TBS has been assigned specific responsibility for leading the overall business transformation and change management agenda.

EMIS is an integrating system and, as such, it cannot drive transformational change but can only provide support to that process. Transformational change is a much larger effort underway within all of TBS.

Within the Expenditure Management Sector, a senior position has now been staffed to co-ordinate end-to-end assessment of business processes and of potential for EMIS to contribute to transformational change.

Business process owners and representatives have now been identified and their roles are formalized within the new EMIS governance structure.

Business owners and designated technical authorities have been identified for each business process or activity area within the scope of EMIS. Technical authorities are assigned to the project as their priority assignment. Designated business owners are responsible for specifying requirements, and are responsible for convening advisory groups to ensure appropriate stakeholder input. Business owners have committed to their participation in the development cycle, notably, product validation testing. Final decisions will rest with business owners.

Testing and quality assurance will be done independently and under formal processes and protocols.

3.2.2 Business Case

Observation

Findings

The current business case is high level, and has not been updated for new initiatives, scope changes or changes in costs to complete as the underlying project management mechanisms to enable these linkages do not exist. There is no link between events during the current phase and how they impact the business case of future phases. With no formal gate process, and no coordination between the business case and project management processes (because they themselves are not formalized), it is not possible to review the implications of project progress on the business case. Furthermore, there is no process for analyzing how these changes to the current phase impact the business case of future phases.

Recommendation

The business case must be kept up to date to reflect changes in the project, and impacts on future phases must be considered.

• A formal process should be implemented to link the business case to the project management metrics that influence it. In addition, there should be a mechanism that ties the resulting updates to the business case to the future phases of the project through the use of sensitivity analysis, steering committee evaluation and approval of the updated business case. This will require previous recommendations related to project management mechanisms to be implemented. 

Management Response

Business cases will be developed for all new project components and undergo a review and approval process and existing business cases will be updated.

• The process now underway, as part of the Action Plan, of determining the scope of the project, developing the Roadmap and subsequently detailed project plans, will lead to the establishment of viable business cases, which can be tracked over time.
• A performance management framework including an outcomes management approach, and key performance measures will support business cases, the Roadmap, and detailed project plans. Key elements and corrective actions include:
  • Following the Treasury Board's Enhanced Management Framework (EMF) and where it is insufficient, consider appropriate, cost-effective and adaptable industry best practices as defined in the Project Management Institute's (PMI) Project Management Book of Knowledge (PMBOK);
  • Support and encourage the development of business cases aligned with Business Transformation Enablement Program (BTEP); and
  • Adapting the TBS CIOB's Performance Model from BTEP.

3.2.3 Accountability of Projects

Observation

Findings

The EMIS Project has multiple clients within TBS representing the Program Sectors and Expenditure Management Sector. The involvement of business process owners is not well defined and acceptance/sign-off procedures are unclear. There have been differing opinions expressed as to who the clients are for this system. There has not been sufficient time to properly gather requirements, develop, test, train and deploy in the earlier iterations of the deliverables. 

Recommendations

Business process owners should formally sign off on deliverables.

• The project needs to clearly establish business process owners, formal acceptance criteria and dedicated representatives from business areas assigned to project team to take accountability for quality acceptance of deliverables. Business management should have ownership of process alignment.
• As the project moves into implementation phases, IT and business management must jointly assess implementation strategies and ensure readiness checks and balances are in place. Project management must develop detailed implementation plans with tasks, dates, deliverables, resources assigned and costs. Readiness assessment checklists should be developed and pre-implementation activities well understood, monitored and executed.

Management Response

For each of the EMIS priorities a business owner has been identified and will be responsible for defining the business requirements, acceptance criteria and quality standards. The project will institute several gates in the development cycle for each initiative to determine both EMIS and the client's readiness to proceed to the next phase and if the solution will still meet the business goals. Readiness assessments (lessons learned) will be developed as the project gains experience in delivery solutions.

3.2.4 Accountabilities of Multiple Stakeholders

Observation

Findings

There is no formal MOU in place with PWGSC that specifies the expectations and requirements of the EMIS project. Senior management feels that TBS owns the system and data, and client departments have to use the tools TBS provides for expenditure management. There is considerable lead-time expected by departments to make changes to coding blocks and charts of accounts to meet requirements of TBS and PWGSC financial reporting through the Central Financial Management Reporting System (CFMRS). TBS requirements and timelines have not been developed and communicated on a timely basis. There is a risk that the departmental reporting process may be impaired during FY 2005/06.

Recommendations

An MOU should be put in place with PWGSC that specifies requirements, timeframes and responsibilities.

• There needs to be improved definition of external stakeholders and their relationship to the project. Improved requirements definition and communication with the departments need to be acted upon. Risks of failing to meet required changes and interfaces with CFMRS and EMIS need to be assessed. 
• Departmental readiness assessments should be conducted for iterations of implementation planned for this phase and Effective Project Approval (EPA) Phase 2.

Management Response

Business process owners - both senior management accountability and the technical process owners - have been identified and the latter will be dedicated to the EMIS team as necessary.

The Senior Information Technology (IT) Executive position will be staffed to lead the IT sections of the project.

A manager responsible for testing and quality assurance will be added and will report to the Project Director. Effective immediately, the testing team will be separated from the development environment and will report to the Project Director.

Agreed implementation plans and strategies will now be included in all EMIS business plans.

The project will assess the best methods to ensure that departments are informed of EMIS initiatives and that they have the capacity to carry out these functions.

3.3 Technology

3.3.1 Common Technology, Tools and Methodology

Observation

Findings

A number of different project methodologies are used on the project, yet there is no roadmap which ties them together to ensure that there are no areas being neglected and that the outputs from one area are compatible with the inputs from another. There is no process in place to ensure that methodology components such as change management and communication are not neglected.

There are no formal tools used for risk assessment and control within the project. Each sub-team within the project has its own approach which does not support a systematic roll-up of this important information.

Recommendations

A standard project life cycle and business transformation methodology needs to be fully developed, training provided and deployed on the project. This must meet the requirements of TBS Management Framework.

The Project Management Office is responsible for methodology documentation and should provide adequate support to the Project Team. The project management office should also ensure policies, procedures and process controls are developed, training is provided and usage is mandatory. Key areas requiring immediate attention to strengthen methods and tools include:

• The project should create the role of methodologist, responsible for ensuring that the methodologies used are documented and that they are tied together in a cohesive and logical fashion. This should include the development of a methodology overview that address the various leading methodologies being deployed on the project (e.g. Rational Unified Process (RUP) methodology, Modified Zachman (data architecture methodology) and IT Infrastructure Library (IT Service Management).
• Proper project gate reviews by business process owners need to be implemented.
• The project management office needs to develop an organizational wide testing methodology where IT and business process owners develop testing requirements. Further, business users are involved in acceptance testing and with IT, provide written sign-off and approval of each testing process.
• Quality assessment procedures should be conducted at key project milestones and gates (e.g. Quality Assessment of technology platform prior to deployment to client departments throughout the federal government).

Management Response

A methodologist will be hired, after a review of all project processes, to compile an integrated methodology and to give some structure for future design and architecture work (Phase II).

Gate reviews will be identified in project plans, including the business cases for individual components.

As mentioned previously, testing will be independent and systematic (but an review unit external to the development domain will report directly to the Project Director and not to the project management office).

Quality Assurance (QA) will be factored into project controls. An internal testing and quality assurance function has been established, reporting directly to the Project Head and regular independent quality assurance reviews will be conducted by outside resources.

A review of IT infrastructure to examine cost-effectiveness, plans, establish a budget, and identify cost-reduction possibilities will be undertaken. Also review of the arrangement for systems development services will be carried out to determine the best way to source needed skills as well as development-related software, and methodologies to determine cost effectiveness and opportunities for savings. 

3.3.2 IT Security Resources for Projects

Observation

Findings

High-level security requirements appear to be understood (e.g. protected environments). However, with no integration of project methodologies, it is difficult to assess the completeness of security planning and implementation within the applications. Security implications are currently dealt with by a number of individuals, but there is no security officer with overall accountability. Security was not well defined in the initial scope definition and was added on during design activities in this phase. 

Recommendation

Security must be given a higher priority on the project.

• An overall methodology roadmap should be developed which highlights security implications and steps. A security officer role should be formalized to take accountability for security.

Management Response

The project has implemented a security framework that has been approved by TBS, CSE and PWGSC to ensure tight co-operation of security requirements and testing.

The project has created an Information Systems Security Officer position. This person will operate at the team management level to ensure compliance with the security framework and with the project performance management plan. First results from external reviews of the EMIS project, relative to progress government-wide, have been positive.

 

• Previous Page
• Table of Contents
• Next Page