Next Generation Fighter Capability: Life Cycle Cost Framework

Acknowledgements

The KPMG Independent Review and associated Framework have been commissioned by the Treasury Board Secretariat as part of its responsibility in the Government of Canada's Seven-Point Plan in response to Chapter 2 of the 2012 Spring Report of the Auditor General of Canada.

These documents have been edited for publication in accordance with the publication and communications quality assurance processes within the Treasury Board of Canada Secretariat. The PDF document is the official version.

Table of contents

1 Introduction

On , the Auditor General of Canada presented his 2012 Spring Report to Parliament. The Government accepted the Auditor General's findings and the recommendation contained in Chapter 2, Replacing Canadas Fighter Jets, in which the Auditor General recommended that the Government refine its estimate for the full life cycle costs of the F-35 and make the estimate public.

In response, the Government announced seven steps to address the Auditor Generals findings and the recommendation. The seven point action plan follows:

  1. The funding envelope allocated for the acquisition of the F-35 will be frozen.
  2. The Government of Canada will immediately establish a new F-35 Secretariat (NFPS) within the Department of Public Works and Government Services Canada (PWGSC). The Secretariat will play the lead coordinating role as the Government moves to replace Canadas CF-18 fleet. A committee of Deputy Ministers (DMGC) will be established to provide oversight of the F-35 Secretariat.
  3. The Department of National Defence, through the F-35 Secretariat, will provide annual updates to Parliament. These updates will be tabled within a maximum of 60 days from receipt of annual costing forecasts from the Joint Strike Fighter program office, beginning in 2012. The Department of National Defence (DND) will also provide technical briefings as needed through the F-35 Secretariat on the performance schedule and costs.
  4. The Department of National Defence will continue to evaluate options to sustain a Canadian Forces fighter capability well into the 21st century.
  5. Prior to project approval, Treasury Board Secretariat will first commission an independent review of DND's acquisition and sustainment project assumptions and potential costs for the F-35, which will be made public.
  6. Treasury Board Secretariat will also review the acquisition and sustainment costs of the F-35 and ensure full compliance with procurement policies prior to approving the project.
  7. Industry Canada, through the F-35 Secretariat, will continue identifying opportunities for Canadian Industry to participate in the F-35 Joint Strike Fighter global supply chain, as well as other potential benefits for Canada in sustainment, testing, and training, and will provide updates to Parliament explaining the benefits.

In order to provide Treasury Board (TB) Ministers, Parliament, and the Canadian public with the estimated life cycle costs of acquiring the Joint Strike Fighter as the future fighter capability for the Canadian Forces, the Treasury Board Secretariat (TBS) commissioned an independent review to inform its analysis and advice regarding the Department of National Defences Next Generation Fighter Capability (NGFC) Project.

The NGFC Life Cycle Costing (LCC) Framework, contained in this report, establishes a clear direction for the calculation and reporting of NGFC LCC. This report is principles-based, prepared for TBS and DND purpose and should be considered in its entirety with particular reference to Section 2.5 – Understanding this Document.

2 Context and Approach

2.1 Purpose

The NGFC LCC Framework establishes a clear direction in the development of costing information for use by officials at DND and other responsible federal departments, by their Ministers, by relevant committees of Ministers (e.g. TB and Cabinet), and by Parliament and its Agents. The NGFC LCC Framework is a principles-based document developed to facilitate accountability, traceability, transparency and consistency over the life of the NGFC. As a result, management judgement is required in the application of the NGFC LCC Framework to support the development of relevant life cycle cost information applicable for the intended purpose.

2.2 Context

The ultimate purpose of life cycle costing is to inform decision making. Life cycle costing, and the NGFC LCC Framework principles, can be applied throughout the project life to support various purpose and decisions. As a result, it is critical that the specific purpose of the cost information is considered when management is applying judgement relating to the LCC principles and boundaries.

Life cycle costing is fundamentally a forecasting activity and is therefore imprecise and uncertain due to the unforeseen impact of various potential future events. The level of uncertainty with respect to cost estimates in early project phases is significantly higher than the cost uncertainty level in later project phases. Given the early Options Analysis phase of the NGFC project, the level of cost uncertainty will by high (see section 3.4) and care must be taken to both estimate and communicate the degree of uncertainty within the LCC estimate.

2.3 International Context

To accomplish the purpose, the NGFC LCC Framework considers current Canadian guidance and requirements. The Framework is based on international leading practices drawn from comparable large scale military initiatives, as well as broader international government initiatives, and leading practices in life cycle costing.

KPMG has derived leading practices through careful consideration of a variety of relevant sources, (outlined in Annex A of this report), as well as through our understanding of common practices conducted by other JSF partner countries, where made available to us (outlined in Annex A of this report). KPMG consulted with 5 JSF partner countries (Australia, Denmark, the Netherlands, Norway and Turkey) through the use of a 17 question consultation memo. Responses were received from Australia, the Netherlands, and Norway. Publically available information from the United States and the United Kingdom has also been reviewed in this report. The practices summarized in this framework are those relevant for consideration in the development of the NGFC LCC. It is important to recognize that not all aspects of LCC leading practices are expected to be fully adopted by all agencies, and for all projects. Consideration of the extent to which leading practice is followed varies based on the nature of the project and cost/benefit considerations.

2.4 Framework Design Considerations

Developing a clear understanding of the purpose of the NGFC LCC Framework is a key design consideration, since it supports the required information for decision making by various users. The following information needs have been noted below:

  • Enable the provision of annual costing updates starting in 2012 regarding the Joint Strike Fighter (JSF) program as outlined in step c. of the 7-point action plan
    • The Department of National Defence, through the F-35 Secretariat, will provide annual updates to Parliament. These updates will be tabled within a maximum of 60 days from receipt of annual costing forecasts from the Joint Strike Fighter program office, beginning in 2012. The Department of National Defence will also provide technical briefings as needed through the F-35 Secretariat on the performance schedule and costs.
  • Ensure that the estimated life cycle costs can be examined through an independent review process as noted in the National Fighter Procurement Secretariat NFPS Governance Terms of Reference
  • The DMGC, the ISC (Interdepartmental Steering Committee) and the Secretariat, along with TBS, the Department of Finance and PCO, will collaboratively ensure that due diligence is performed throughout the implementation of the NFPS action plan as follows: The provision of independently verified annual costing updates regarding the Joint Strike Fighter program.
  • Seek TB approval (contracting and project authorities) through the period of the NGFC Project which includes fulfillment of step e. of the 7-point action plan
  • Prior to project approval, Treasury Board Secretariat will first commission an independent review of DND's acquisition and sustainment project assumptions and potential costs for the F-35, which will be made public.
  • Enable TBS to review the acquisition and sustainment costs of the F-35 as per step f. of the 7-point action plan
  • Treasury Board Secretariat will also review the acquisition and sustainment costs of the F-35 and ensure full compliance with procurement policies prior to approving the project.
  • Although not stated specifically in reference documents, the needs listed below were also noted in consultations with key stakeholders as specific requirements of the NGFC LCC Framework:
  • Seek updated Cabinet approval for NGFC – during the NGFC initiative, it is possible that there will be a requirement for additional approvals from Cabinet
  • Identify incremental budget requirements for DND
  • Identify contracting requirements
  • Identify the context and funding envelope within the portfolio of the Canada First Defence Strategy.

As noted in the purpose statement, key framework design objectives include:

  • Accountability
    • To Cabinet and Treasury Board and the officials supporting the accountable Ministers in the approvals and oversight of the NGFC Project
    • To Parliament, its Committees and its Agents to enable these organizations to fulfill their mandates
  • Traceability
    • To relate the overall capability of the NGFC in a clear manner to the summary and detailed elements of the costing framework
    • To relate the changes in Life Cycle Costs to specific changes in capability, risks, assumptions and other elements of the NGFC LCC
  • Transparency
    • At both a sufficient level of detail as well as at appropriate summary levels, to allow stakeholders to adequately understand and, as required, to validate the information
    • Between other JSF Countries, to assess comparability and/or to understand the context of different figures being quoted from different countries
  • Consistency
    • Over time, as estimates are refined from year to year and the risks and uncertainties inherent in NGFC are addressed
    • Across the various approval mechanisms (e.g. Cabinet approvals, TB contract approval, TB project approval, TB expenditure approvals, DND internal approvals, etc.).

2.5 Understanding this Document

This document is intended to assist TBS and DND in their preparation of a Life Cycle Cost (LCC) estimate of the Next Generation Fighter Capability Program (Program).

KPMGs procedures consisted solely of inquiry, comparison and analysis of identified and provided information and relevant information from comparable jurisdictions. We relied on information provided by project participants without verification. The information contained in this document does not constitute an audit. Accordingly, KPMG does not express an opinion on such matters.

This document has been prepared for TBS and DND purposes to support DNDs LCC estimate of the Program. No one should act on the information within this document without conducting additional analysis. The application of the NGFC LCC Framework will be completed by DND, and DND is responsible for decisions made in regards to its application.

This document should be considered in its entirety. Selection of, or reliance on, specific portions of this document could result in the misinterpretation of comments and analysis provided. We will not assume any liability in connection with the reliance by any third party on this document.

We reserve the right, but will be under no obligation, to review all findings, conclusions and calculations included or referred to herein and, if we consider it necessary, to revise our findings, conclusions and calculations in light of any information which becomes known to us after the date of this document.

3 Life Cycle Costing

Life Cycle Cost is an estimate of all costs associated with a product, project or program over its life cycle. This term is further defined below. Life Cycle Costing is the process through which a life cycle cost is developed. The Life Cycle Costing approach may not aim to develop cost estimates for all elements of a project or program, nor for the entire life of a project or program. The focus of Life Cycle Costing is to develop cost estimates that are driven by the purposes of the decision maker – they can take a very macro view–"what is the cost of the NGFC over its entire life"–as well as a very micro view–"which fuel pump design is the most cost effective".

3.1 What is a Life Cycle Cost Estimate (LCCE)?

There are a number of definitions for a Life Cycle Cost Estimate. For the purposes of this document, we will use the definition of Life Cycle Cost Estimate (LCCE) taken from the United States General Accounting Office (GAO) Cost Estimating and Assessment Guide1, supported by Figure 1 provided by ISO 15686-5:

A life-cycle cost estimate provides an exhaustive and structured accounting of all resources and associated cost elements required to develop, produce, deploy, and sustain a particular program. Life cycle can be thought of as a "cradle to grave" approach to managing a program throughout its useful life. This entails identifying all cost elements that pertain to the program from initial concept all the way through operations, support, and disposal. An LCCE encompasses all past (or sunk), present, and future costs for every aspect of the program, regardless of funding source.

Life-cycle costing enhances decision making, especially in early planning and concept formulation of acquisition. Design trade-off studies conducted in this period can be evaluated on a total cost basis, as well as on a performance and technical basis. A life-cycle cost estimate can support budgetary decisions, key decision points, milestone reviews, and investment decisions.

Figure 1: Whole of Life and Life Cycle Cost
Whole of Life and Life Cycle Cost. Text version below:
Figure 1 - Text version

This figure is block diagram in four layers. The top layer is labelled "whole-life costs" which is broken into three sub components at the second layer: Income, Life-cycle Cost and Externalities. The middle block, life-cycle costs is further broken down at the third level into four sub-components: development, acquisition, sustainment and operations, and disposal. The final layer is titled Environment, and underlies the entire diagram.

As shown in the diagram above, Life Cycle Costing can be considered as a component of whole of life costing. Under this framework, income (e.g. salvage costs during disposal phase) and externalities (e.g. costs or benefits impacting outside parties, including other government departments) are not incorporated into the LCCE. If any revenues are to be considered in the assessment of the NGFC, these should be considered through other processes, with the LCC forming a key input.

Our research shows that when performing Life Cycle Costing it is typical to separate the life of an asset into key phases that reflect unique aspects of the program's life. Within the NGFC Project Development these typical phases have been further refined into sub-activities. The NGFC Project is currently in the Options Analysis Stage, which is part of the Development Phase. The key phases of the Life Cycle (as depicted in Figure 1) can be described as:

Figure 2: Life Cycle Phases
Typical Phase Phase Description
NGFC Project Development All activities necessary to achieve expenditure approval.
NGFC Acquisition All activities necessary to introduce the NGFC into operational service.
NGFC Sustainment & Operations Ongoing operations and maintenance of the NGFC aircraft.
NGFC Disposal Removal of NGFC from service and retirement of any potential financial liabilities.

The following diagram is provided to give clarity on the relationship between these phases and those used within the Government of Canada and DND. As of the date of this report, the NGFC Project was in the Options Analysis phase.

Figure 3: Life Cycle Phases Compared to Government of Canada Phases
Life Cycle Phases Compared to Government of Canada Phases. Text version below:
Figure 3 - Text version

This figure has two rows of angled blocks. The first row consists of four large shaded blocks which reflect each of the key phases of life cycle costing: Development, Acquisition, Sustainment and Operations and Disposal. The second row is listing of steps associated with the key phases. Directly below Development are the steps Identification, Options Analysis and Definition. Below Acquisition are the steps Implementation and Close out. Below Sustainment and Operations is the step Sustainment and Operations. Finally, below Disposal is the step Disposal.

It should be noted that these cycles will often necessarily overlap. For example, the introduction of the NGFC would be staged over a period of years such that a number of aircraft would be operational while other aircraft are still being acquired.

LCC analysis can be carried out at any stage within these Phases. As the NGFC evolves through these stages, the model2 also evolves in a traceable fashion. The changes would include improvements in fidelity of estimates through to the inclusion of actual data. The model must either be adaptable to meet such future requirements, or the information must be able to be transparently and traceably transitioned to another suitable model within the overall financial framework of DND.

3.2 What is being costed?

The Life Cycle Costs must focus on costing those elements that are within the prescribed boundary of the program.

In the NGFC context, it is fundamentally important that the projects LCC model outputs are traceable back through this golden chain of logic (Figure 4), starting with the 'Canada First Defence Strategy'3.

Fighters

Starting in 2017, 65 Next Generation fighter aircraft to replace the existing fleet of CF-18s. These new fighter aircraft will help the military defend the sovereignty of Canadian airspace, remain a strong and reliable partner in the defence of North America through NORAD, and provide Canada with an effective and modern air capability for international operations.

In the absence of this chain of logic, it is feasible that the effective program being costed would not align with the endorsed requirements of Government. One of the critical components, therefore, of the NGFC Life Cycle Costing is the analysis and assessment of both the project and program requirements and the applicable costs.

Figure 4: Golden Chain of Logic
Golden Chain of Logic. Text version below:
Figure 4 - Text version

This figure consists of four shaded blocks. The first block Canada First Defence Policy, the second block is Statement of Requirements, the third block is Functional Specifications and the fourth, last block is Life Cycle Costs.

The seven point action plan (summarized in Section 1) notes that the Department of National Defence will continue to evaluate options to sustain a Canadian Forces fighter capability well into the 21st century. However, the specific scope of work of this current study is to detail the framework to be used to assess the Life Cycle Cost Estimate (which includes aspects such as development, acquisition, sustainment, upgrade, operation and disposal/decommissioning) of a fleet of 65 F-35A Lightning II Joint Strike Fighters, including propulsion and mission software systems, throughout its expected operational life.

3.3 Scope Considerations of the NGFC LCC

The Definition of a Life Cycle Cost Estimate provided in Section 3.1 indicates "A life-cycle cost estimate can support budgetary decisions, key decision points, milestone reviews, and investment decisions". In developing a framework that can support life cycle costing, one must consider the current and future decisions that would require LCC information.

3.3.1 Hierarchies of Cost Elements for the NGFC LCC

The broad LCC purposes, throughout the life of the NGFC consider the following:

  • Contract – What is the cost of the Acquisition and Sustainment contract(s) for NGFC?
  • Project – What are the total project costs to deliver the NGFC to Canada?
  • Incremental – What are the increased costs to operate the NGFC above the current Fighter Capability?
  • Program – What are the total program costs to deliver and operate the NGFC through to its withdrawal from service?
  • Portfolio – What are the other related activities and costs, beyond the NGFC, that the NGFC Program relies upon?

To a large extent, the applicable boundaries of the LCC model maintained within DND would depend on the requirements of key stakeholders and as a result, would need to be adaptable for these requirements. These requirements and the adaptation of the LCC model are discussed in more detail below, and in section 3.3.2 of this report.

The boundaries of scope are established to reflect three key aspects: the different purposes of the LCC, the different information requirements related to those purposes, and the different cost elements relevant to those purposes. In clarifying the different purposes, the concepts of accountability and roles in decision making and approvals are also considered.

Contract level

This level is focused on the specific contracts that would be required to deliver the capabilities of the NGFC Program overall. It includes contracts that would be part of the NGFC Project (see below), and contract costs related to the NGFC Program relating to ongoing sustainment and operations after the project itself has completed. After establishing contract requirements through consultation with DND, contract approval is sought by the Minister of Public Works and Government Services Canada through Treasury Board.

Project Level

A project is defined4 as an activity or series of activities that has a beginning and an end. A project is required to produce defined outputs and realize specific outcomes in support of a public policy objective, within a clear schedule and resource plan. A project is undertaken within specific time, cost and performance parameters. Project start is normally taken to be at the end of the Identification Phase as the project begins the Options Analysis Phase. Project completion and Close Out is normally taken as when the Full Operational capability is achieved. These definitions are reflected in existing DND documentation5.

The NGFC Project has been underway within DND (as per the definition in the previous paragraph) we understand it has not yet required to be submitted to Treasury Board for approval. When it does come forward there would be an increasing depth of analysis and other project aspects such as clarity on the exact nature of the boundary between the ending of the project and the transition to ongoing NGFC Program operations.

NGFC Project costs would include: acquisition costs of the aircraft, initial logistics support, along with all other costs directly6 related to the introduction of the NGFC capability for Canada. A portion of the contract costs would thus be included here (i.e. the costs related to the project). The project level costs would also include: contract management costs, initial training, infrastructure, studies and analysis costs and project management costs. Being in the Options Analysis Phase necessarily limits the expenditures on the overall activities and would also then limit the depth and extent of detailed costing information at this stage.

The Project level focuses on the total resources under the control of the NGFC Project Office, along with their accountability to Deputy Minister of National Defence for the delivery of the project components of the NGFC, including major decisions to manage the project through to successful completion. Project Approvals and Expenditure Authorities at various stages would be sought by the Minister of National Defence through Treasury Board.

Incremental Level

Incremental costs are the changes in resources above or below the costs for the existing Fighter Capability Program required over the life of the capability (i.e. post the completion of the NGFC Project). This could include items such as increased support costs, increased fuel costs and additional personnel costs. Incremental level costs are distinct from the Project level costs because these are the incremental costs that extend beyond the life of the project.

As part of the Project Approval process the Minister of National Defence would inform Treasury Board of the overall and annual estimated incremental costs associated with the life of the Capability following an appropriate basis of accounting. Following Project completion, approvals may also be subsequently sought to adjust Annual Reference levels based on operational experience and circumstances which become clearer over the operating phase of the NGFC.

Program Level

This level focuses on the full costs of all DND capabilities that directly contribute to the Fighter Capability Program. A program7 is any group of resources and activities, and their related direct outputs, undertaken pursuant to a given objective or set of related objectives and administered by a department or agency of the government. Distinguished from a project, which has a specific objective, activity, beginning and end, a program may include various projects at various times.

For clarity, the Fighter Capability Program is seen as being made up of two portions. Currently, the fighter capability is provided through the existing CF-18 fleet. In the future the fighter capability would be provided through the NGFC. These costs would include all the costs in the contract, project and incremental level along with existing costs, such as, the squadron operating and sustainment costs, facilities costs associated with their operations and personnel related costs.

Program costs are made up of the existing program costs for the CF-18 capability with the addition of the costs of the Incremental level and the Project level noted above. These costs would include such items as the Squadron operating and sustainment costs, facilities costs associated with their operations and personnel related costs.

NGFC Program Cost = NGFC Project Cost + NGFC Incremental Sustainment Costs + NGFC Incremental Operating Costs + Current (CF-18) Fighter capability costs

Figure 5: Fighter Capability Program, Project and Incremental Costs
Fighter Capability Program, Project and Incremental Costs. Text version below:
Figure 5 - Text version

This diagram is a line graph which depicts the Total Fighter Capability Program Costs. The y-axis depict the amount and the x-axis depicts time. The line is horizontal at first, then rises, levels off then drops down again to a new horizontal which is above the original level.

The first horizontal level is extended through the entire graph area. Underneath are two shaded areas of equal size, titled "Current Fighter (CF-18) Capability Program Costs" followed by "NGFC Capability Program Costs". These areas meet in a diagonal line, showing overlap of the two costs.

The area between the extension of the original horizontal line and the cost lines rise, level off and drop is shaded and labelled "NGFC Project Costs". This area overlaps with the area under the final horizontal level of the line, which is titled "NGFC Increment".

The relationship among the program, incremental and project costs is depicted in Figure 5 above. The Contract level has not been depicted as contract costs would contribute to both project and incremental costs. Note also that Figure 5 is not intended to be to scale in terms of the relative sizes of program costs, project costs and incremental costs, and Figure 5 depicts positive incremental cost for illustrative purpose only.

Portfolio Level

The Portfolio level includes the costs from all previous cost levels, and the addition of costs of other capabilities that indirectly contribute to the NGFC, based on sound judgment8. Examples of these types of costs are the recruiting systems that provide basic training to future pilots, as well as surveillance and intelligence equipment that may compliment the effectiveness of the NGFC.

Summary

Each of these boundaries becomes important at critical times in the decision making process, as well as aspects of ongoing management. This is examined below and in more detail in section 4.3.2 of this report.

NGFC LCC Framework: Products

The Life Cycle Costing process establishes a common and consistent approach to providing products to key decision makers and stakeholders. The process ensures that information provided for differing purposes can be reconciled to help ensure consistency in practice and reporting.

In this context, the products refer to what is provided to the final decision maker. These products are generally tailored for a non-expert audience and often form part of a suite of documents being provided to support decisions.

In the context of the NGFC LCC Framework, some of these products would also segment the costs against the Parliamentary Vote Structure for DND (i.e. into Votes 1, 5 and 10, representing Operating Expenditures, Capital Expenditures and Grants and Contributions)

These products would also directly support the approvals needed for decision points such as Contract, Project and Expenditure approvals.

Figure 6: NGFC LCC Framework
Next Generation Fighter Capability Life Cycle Costing Framework. Text version below:
Figure 6 - Text version

This diagram is a three-dimensional cube having specific volumes associated with figures representing costs.

The dimensions of the cube are time (length), work breakdown structure (height) and Votes (depth). The Time dimension (length) has many partitions and along the axis is labelled Year 1, just less than half-way is labelled year 10 and at the end, is labelled Useful life. The Votes dimension (depth) has three partitions, labelled Vote 10, Vote 5 and Vote 1. The Work Breakdown Structure dimension (height) has many partitions but none are labelled.

The document figure labelled "LCC for Programming" is associated with approximately half the cube (year 1 to before year 10, all work breakdown structure and all votes).

The document figure labelled "LCC for Useful Life Analysis" is associated the entire cube (all years, all work breakdown structure and all votes).

3.3.2 Decisions Requiring Use of the NGFC LCC Framework

The following section outlines the specific milestones or decision points in the coming years that would require the use of the NGFC LCC. These are based on the existing federal government approval processes including Departmental decisions and approvals, Treasury Board and Cabinet approvals and Parliamentary approvals.

This section does not attempt to address the approvals obtained and activities undertaken prior to the development of this NGFC LCC Framework. DND has completed much of the first two phases of a typical project, namely the Identification Phase and the Options Analysis Phase.

3.3.2.1 DND Approvals

Within DND, the NGFC LCC Framework would be used to support the full scope of strategic and operational planning and management activities. During the Development Phase the LCC Estimate would be progressively improved through the inclusion of more reliable information and greater certainty around a number of aspects of the project. The model would be routinely updated as new information becomes available. Leading up to decisions points the model would be independently reviewed to provide assurance to decision makers that the LCC Estimate is suitable given the stage of the program and the programs overall characteristics.

Through the Acquisition Phase more and more actual data would be available, as well as better data on operating and sustainment costs. Through Project Definition and Implementation, greater focus would be on using the LCC project costs in support of project management and performance management approaches Annually the LCC Estimate would likely be reviewed and updated to support the improved understandings of project elements as well as supporting revisions to the operating and support costs to assist DND budgeting.

3.3.2.2 Project Approvals

Prior to commencing the Definition Phase of a project, Treasury Board Project approval must be sought for a project of this magnitude9. The scope, schedule and cost of the project must be outlined in a Project Brief as part of the submission to Treasury Board by the Minister of National Defence in seeking Expenditure Authority. A Project Brief includes a synopsis of the Project Management Plan, the Project Charter and the Business Case. The specific costing information requirements include guidance that the costing analysis includes all life cycle costs10. The following paragraphs outline more specific direction on this costing analysis.

The Project Approval approach recognizes that the degree of certainty increases as the project progresses. There are three broad levels of cost estimates that are used in this approval process11:

  • Rough Order of Magnitude (ROM) estimate: Preliminary estimate generally based upon assumptions and cost estimates derived from previous experience and very high level costing approaches. As a result, generally large variability between the cost estimate and actually cost will exist. This is generally the type of estimate that would be available during the Identification and Options Analysis phases of a project.
  • Indicative estimate: A developed estimate supported by further cost breakdown definition, application of standardized costs and other research to support assumptions and cost estimates.
  • Substantive estimate: A highly detailed cost breakdown of planned expenditures for all significant cost elements. The actual cost of the project or program is expected to be within a much smaller band of values as the degree of certainty in cost estimates increases.

Prior to seeking the first Expenditure Authority and Project Approval from Treasury Board, a project would be expected to have only ROM estimates12. As the project approaches the Definition stage, cost estimates would need to improve beyond the ROM estimates. Expenditure Authority for the Definition Phase would be sought for the first portion of the project at this first TB Submission. The Definition Phase clarifies the work to be done in Implementation Phase(s) to allow for the preparation of a substantive cost estimate to be developed for the Implementation Phase(s). In this initial Project Brief and submission, there would be substantive costs for this first portion of the project along with an indicative estimate of total project costs. In addition, the full amount of ongoing costs, over the timeframe of the NGFC Program, would be presented as information to TB Ministers along with a statement as to how incremental amounts relative to existing program costs were being funded (e.g. reallocation).

As part of the Project Brief referenced above, the Minister of National Defence would also propose milestones at which the project would return to Treasury Board and when further Expenditure Authority would be sought. For the subsequent submissions (e.g. when DND was about to begin the Implementation Phase), DND would seek Expenditure Authority for the part of the Implementation Phase for which it has substantive costing. This submission would include a revised Project Brief to support the request. The costing in the Project Brief would include updated project costs and would also be accompanied by updated NGFC incremental costs. If the project scope or costs have changed from what was approved at the Project Approval decision point, a Revised Project Approval decision must also be sought. The submissions for Expenditure Authority (and updated Project Authority, if required) for the NGFC Project would require both Project Level and Incremental Level information.

To be effective, the LCC data presented to Treasury Board in the various submissions must represent the total Project and Incremental Costs necessary to deliver the endorsed Statement of Requirements. To provide the necessary context, it would be desirable to present the complete Life Cycle Cost picture; that is the total NGFC Project Costs, the NGFC Incremental Costs along with the total NGFC Capability Program Costs.

3.3.2.3 Contract Approvals

A separate Treasury Board submission (or joint submission with the Project Approval and Expenditure Authority Submission noted above) is required to obtain Contract Authority to enter into a contract for the NGFC. Contract Authority would be sought by the Minister of Public Works and Government Services Canada. The scope of the costs for the contract would be for the acquisition costs and/or the sustainment costs depending on the details of the procurement approach. Depending on the contract length, the sustainment costs could also extend beyond the completion of the project through to the operations phase of the NGFC Program.

If the contract costs subsequently increased and/or there were contract amendments to increase the contract value, scope and/or timeframes additional Treasury Board submissions would be required. In each submission relating to Contract approval, the NGFC Project (and program) would require contract level information from the NGFC LCCE.

3.3.2.4 Accountability and Reporting

DND is required to report to Parliament on the NGFC, as part of the DND Annual Update. The primary purpose is to provide Parliament with an understanding of the progress of the NGFC and an update on the life cycle costs. As this Annual Update should be promoting transparency and accountability the scope of the update should include multiple levels within the NGFC LCC Hierarchy. Specifically the update should include all costs out to and including the program level. The span of years for the project reporting should include all estimated costs out to the disposal of all the NGFC as well as already incurred costs.

3.4 Fundamental Limitations of Life Cycle Costing

Figure 7: Cone of Cost Uncertainty
Cone of Cost Uncertainty. Text version below:
Figure 7 - Text version

This diagram is a line graph where the horizontal axis is labelled knowledge/time. The start of the horizontal axis is labelled "Conception", and the end of the horizontal axis is labelled "Disposal". The vertical axis is not labelled but has a specific segment labelled "Cost Uncertainty". The top and bottom of the Cost Uncertainty segment are the starting points of the two lines on the graph, which, from the bottom curve upward and from the top curve downward toward each other, finally meeting at a point that is directly above "disposal" on the timeline. The area between these two lines is a cone shape.

"the probability of a cost estimate becoming reality, when expressed as a single number, is precisely zero"
NATO RTO-TR-SAS-076

Life Cycle Costing is fundamentally a forecasting activity and is therefore imprecise, uncertain and highly sensitive to many factors that may be difficult to quantify at the time when the LCC is developed. As a project matures, knowledge would increase as calculations are based on increasingly detailed and reliable data and assumptions would be tested and validated, leading to more certainty around Life Cycle Cost estimates.

Government and the public often seek a higher degree of precision than can be justified with the information available. Care needs to be taken to both estimate the degree of uncertainty (confidence level) and to communicate this in a clear manner.

Over the lifespan of the initiative, the uncertainties in the costing would diminish. For example, during the Development Phase, the uncertainty related to subsequent phases (e.g. Acquisition and Sustainment or Operations or Disposal) would remain relatively high, as represented by the far left side of Figure 7. As the Development phase continues and nears completion, some of the costs of the later phases would become more certain. However, there would continue to be significant uncertainty in the estimates related to Operations until there is actual experience in the Operations phase. As the Operations phase continues, costs would be more certain, but even then some uncertainty would remain related to, for example, fluctuations in fuel costs or currencies.

3.5 NGFC LCC Framework Evolution

The NGFC LCC Framework has been developed to provide clear direction for the development of LCC information over the life of the NGFC. As a result, while the NGFC LCC Framework remains relatively stable throughout, the LCC information and underpinning data, assumptions, etc. would evolve as it is refined throughout the programs life. As a result of these refinements, decision makers and stakeholders would have increasing confidence in the costs presented as the program progresses.

As an example of Framework evolution, within the Australia Defence Materiel Organization, the expectations of increasing levels of project knowledge are captured through the Project Maturity Score. The Project Maturity Score is derived from the study and analysis of seven key attributes of a project:

  • Schedule – What confidence do we have in the schedule?
  • Cost – What confidence do we have in the project cost estimate?
  • Requirement – How well is the requirement defined and understood?
  • Technical Understanding – How well do we understand the solutions?
  • Technical Difficulty – What is the technical complexity in delivering the solution?
  • Commercial – What confidence do we have that industry can deliver the solution?
  • Operations and Support – Is the effect on the operating and support environment understood and planned?

The Australian Defence Materiel Organizations expectation is that, as the project matures and passes through its internal and external reviews, knowledge and confidence in each area would increase. Each of these above attributes is scored on a range of 1 through 10 so that the maximum score is 70. As illustrated in the Figure 8 below the projects score should increase over time.

Figure 8: Notational Project Maturity Scores 13
Notational Project Maturity Scores. Text version below:
Figure 8 - Text version

This diagram is a line graph titled "Benchmark Project Maturity Scores at Lifecycle Gates".

The horizontal axis, which indicates life cycle gates, is labelled using two rows of information. The Capability Definition label overlaps with 5 partitions of the horizontal axis in order as follows: Enter DCP, Decide Viable Capability Options, 1st Pass, Industry Proposal/Offers and 2nd Pass. The Capability Delivery label overlaps with the 2nd Pass partition from the Capability Definition Label and with 8 other labels in order as follows: Enter Contract, Preliminary Design Review, Critical Design Review, System Integration & Test, Acceptance Testing, Acceptance Into Service, Final Contract Acceptance and Project Completion.

The vertical axis, which indicates benchmark project maturity scores is partitioned in bands as follows: 7 to 13, 14 to 20, 21 to 27, 28 to 34, 35 to 41, 42 to 48, 49 to 55, 56 to 62, 63 to 69 and 70.

The line on the graph rises from the lower left to the upper right, showing specific relationships between lifcycle gates and benchmark project maturity scores as follows: Enter DCP to a score of 13, Decide Viable Capability Options with a score of 16, 1st Pass with a score of 21, Industry Proposal/Offers with a score of 30, 2nd Pass with a score of 35, Enter Contract with a score of 42, Preliminary Design Review with a score of 45, Critical Design Review with a score of 50, System Integration & Test with a score of 55, Acceptance Testing with a score of 57, Acceptance Into Service with a score of 67, Final Contract Acceptance with a score of 69 and Project Completion with a score of 70.

One would expect to see the same maturation of the NGFC LCC Model and related confidence. At the Development phase of the project, little is known and the costs are highly uncertain. As the project progresses through the Acquisition, and Sustainment and Operations phases, more and more information is gathered creating more certainty in the LCC estimate.

It should be clear that, as the NGFC Program progresses; there would be increased confidence in the estimate derived from increased confidence in the underlying data. The maturation of a cost model can also be explained by the maturation of the cost methods used starting with analogous data and completed by using engineering data, supported by actual contract costs. Figure 9 below is presented to provide the type of changes that might occur. Note that this table is highly illustrative and designed to convey the concepts discussed and should not be used to assess any aspect of the NGFC LCC Framework.

Figure 9: Generalized view of increasing project/program knowledge 14
Highly generalised view of how information may change over the course of a project.
  To support Identification To support Options Approval To support initial project approval To support final project approval
Assumptions and Ground Rules Small number of unendorsed global Small number of endorsed global assumptions Large number of endorsed assumptions Smaller number of endorsed assumptions
Cost Breakdown Structure Level 2 - 3 Level 3 - 4 Level 4 - 6 Level 4 - 10
Cash Flow Analogy/Parametric Analogy/Parametric Draft high level Schedule Detailed Schedule
Acquisition Cost Estimates Analogy/Parametric Analogy/Parametric Parametric/Analogy Engineering/Parametric
Operating Costs Heuristics Heuristics/Analogy Parametric/Analogy Parametric/Engineering
Contingency 30% - 50% 20% - 40% 10% to 30% 5% - 20%

The result of this increased knowledge would be increased confidence in the NGFC LCC estimate as the NGFC Project and program progress. This would manifest itself in reduced uncertainty around the baseline costs. This is illustrated in Figure 10 below.

Figure 10: Illustrative example increased cost confidence
Illustrative example increased cost confidence. Text version below:
Figure 10 - Text version

This diagram is a line graph showing three superimposed parabolic curves. The vertical axis of the graph is labelled "Probability of Cost", and the horizontal axis is labelled "Cost".

The first parabolic curve, labelled "Options Analysis" is the widest and the flattest.

The second parabolic curve, labelled "Initial Project Approval" is narrower, and taller than the first.

The third parabolic curve, labelled "Final Project Approval" is the narrowest and tallest of all three.

All three curves reach their highest point (highest probability) at the same cost.

There is a label at the bottom of the graph which states "Illustration of increased confidence in costs, risk management and project maturity leads to reduced uncertainty. Note that in reality, these curves are likely to shift left or right around the true costs that will only be known at project closure/disposal.

4 NGFC LCC Framework

4.1 LCC Framework Components

Figure 11: LCC Framework Components
Life Cycle Costing Framework Components. Text version below:
Figure 11 - Text version

This figure is a sequence of six linked blocks with the caption "People and Organization". At the top, the block is labelled "NGFC LCC Planning" This block leads to the second, labelled "Boundaries and Assumptions". This second block leads to the third block, which is labelled "Develop Model" and then on to the fourth blocks "Data, Populate and Document model". The fourth block leads to the fifth block titled "Review, analyse and update", which leads to the final sixth block, "Interpret and report results", which leads back to the fifth (Review, analyse and update) and fourth (Data, populate and document model) blocks.

A successful LCC Framework and approach requires many interrelated elements, including a team approach, recognition of the sensitivity of outputs to the level of detail and certainty around the cost and time inputs, and the recognition of the inherent limitations of the LCC techniques employed. The NGFC LCC framework is designed to assist NGFC meet the outcomes required by multiple decision makers and stakeholders. The key outcomes are to provide relevant LCC information products, consistent with using reasonable data sources available at the time, to inform budgeting and financial based decisions. The NGFC LCC Framework consists of three components that would support the achievement of these outcomes:

  • People and Organization
  • Processes
  • Products.

For each of these three components, this report summarizes the leading practices, highlights existing Canadian federal government direction/requirements, and provides a summary of the key NGFC LCC Framework principles. As stated previously, the NGFC principles are based on international leading practices, considering Government requirements where applicable.

KPMG has derived leading practices through careful consideration of a variety of relevant sources, (outlined in Annex A of this report), as well as through our understanding of common practices conducted by other JSF partner countries, where made available to us (outlined in Annex A of this report). The Government direction summarized in the following sections has been based primarily on applicable Treasury Board policies and existing DND cost guidance. Treasury Board issues a range of policy instruments that are designed to establish mandatory requirements (rules) or voluntary best practices. There are three types of mandatory instruments (policies, directives and standards) and two voluntary instruments (guidelines and tools).15 The primary TB policy instruments related to the NGFC LCC Framework are found within the Policy Framework for the Management of Assets and Acquired Services and include the Policy on Investment Planning – Assets and Acquired Services, Policy on the Management of Projects, and the Contracting Policy. In terms of guidelines (i.e. a voluntary instrument), the TB Guide to Costing16 is an available resource on costing methodologies. In addition, while not a mandatory policy, DND produces a Costing Handbook that is relied upon as a guide for the preparation of costing estimates17. DND has identified the DND Economic Model, DND Cost Factors Manual, and DND Project Approval Directive as additional documents that provide significant advice and guidance to costing efforts.

4.1.1 People and Organization Considerations

Life Cycle Costing is undertaken by people, working within an organizational structure that will either support or impede their work. As a result, the people and organizational component is foundational to the development of robust and reliable costs. The LCC framework must address key elements around the people, organization, processes and tools to help ensure ongoing and reliable LCC support to the NGFC over the life of the project and program.

4.1.2 Process Considerations

NGFC Life Cycle Costing would require those developing and supporting life cycle costing to follow a set of applicable internationally recognized processes and practices. The following categories summarize the leading practice LCC processes to support the NGFC:

  • NGFC LCC Planning
  • Establish the boundaries/assumptions of the NGFC LCC Model
  • Develop the structure of the NGFC LCC Model to complete the study
  • Establish and document the data and populate the NGFC LCC Model
  • Review, analyze and update the NGFC LCC results.

4.1.3 Product Considerations

The purpose of LCC is to interpret and report relevant life cycle cost information into final LCC deliverables for key decision makers and stakeholders. These deliverables are the tangible outcome of the NGFC LCC approach and the success of the framework would largely be judged by the quality of these deliverables.

4.2 People & Organization

This element addresses the people who would be required to undertake the NGFC LCC. It examines what skills and resources they require to deliver the objectives of LCC. This element also examines important characteristics of the organization within which the cost modeller is managed and supported. The people and organization requirements would change as a project progresses, and based on the characteristics of the project.

4.2.1 Leading Practices

Leading practices within People and Organizations relate to the following key areas:

  • Cost modellers are drawn from a centralized cost estimating organization
  • A learning organization is established, including researching methods to enhance cost estimation and models for life cycle costing
  • The organization supports cost estimation and non-advocate independent review/estimation within the Defence Organization
  • All LCC models/estimates are prepared by suitably experienced personnel
  • Organizationally endorsed, standardized LCC tools/templates are used that are able to be tailored to the specific program
  • Costing follows a standardized and repeatable process managed by the Costing organization
  • Future LCC data requirements are anticipated and data collected accordingly
  • The cost model is developed by multidisciplinary teams.

4.2.2 Current DND/TB Requirements

Although some guidance on People and Organization is available in the TB Guide to Costing, this is not at the level of a mandatory policy instrument. The Guide to Costing refers only peripherally to the People and Organization component, noting the broad consultation required under the principle "Costing requires consultation and judgment".

DND requires formal cost validation using an authorized Subject Matter Expert within the Assistant Deputy Minister (Finance and Corporate Services) organization for all investment projects over $5 million.18

The DND Costing Handbook outlines a cost validation process instituted to examine the financial and strategic aspects of major capital projects and submissions. The purpose of the process is to "be able to attest with a higher degree of assurance"19.

4.2.3 Key NGFC LCC Framework Principles

The Cost Estimating Organization
Organizational Positioning

Drawing the NGFC LCC cost estimators from a centralized organization, that supports their own ongoing learning and continuous improvement, can further enhance LCC capabilities and quality. In this way the LCC estimates would be prepared by personnel skilled and experienced in cost estimation, drawn and supported by a centralized organization. These cost estimators themselves may have a range of professional qualifications, specializations and/or experience, such as, economics, scheduling, engineering, accountancy, mathematics, major equipment or services logistics, project management or other sciences. Cost estimators would be responsible to the Project Manager for the development of the cost model, but accountable to the central organization for its quality.

The advantages of the more centralized management of costing individuals and the organization include:

  • better sharing of resources across the program
  • ability to tap into the resident experts for specific advice
  • facilitates the use of common methods and procedures and commonality and consistency of tools and training
  • career path with more opportunities for advancement
  • the team includes experienced and cost estimators in training who would be better able to be assisted centrally by the resident experts
  • includes persons experienced in the programs major sector (air, land, etc)
  • there is more flexibility in changing the composition of the team as the project progresses and more resources are needed to make it commensurate with the task.
Key Organizational Functions

The LCC organization is made up of two key functions: LCC Estimation and LCC Assurance. Estimation involves the development of the LCC, while Assurance involves independent review of the LCC and may include the development of an Independent Cost Estimate (ICE) as part of the assurance process.

The LCC assurance function is a key component to providing decision makers with the confidence that the LCC estimates submitted are realistic, comprehensive and consistently presented.

Under leading practice, the cost assurance team members undertaking a review of the independent cost estimate would not have any ongoing and or previous involvement within the program under review. This ensures independence and supports transparency of process and outcome.

In order for the work of the cost assurance team to gain credibility, the cost assurance team members must be recruited from those organizationally recognized as the most experienced and knowledgeable on cost estimation; that is the departments acknowledged subject matter experts. The cost assurance team outputs need to be of high quality in order to gain their acceptance internally and externally. It should over time evolve into a recognized centre of excellence.

Other countries have managed LCC Estimation and Assurance through a range of approaches. Full organizational separation occurs within the United States – with Cost Assessment & Program Evaluation sitting independently of project costing teams. Within Australia, Cost Analysis Branch provides independent assurance but sits within the Capability Development Group that also develops cost estimates. In the United Kingdom, Cost Analysis & Assurance Services provides both cost estimation and cost assurance within the same organization, but using policies and procedures to manage any perceived conflicts.

Ideally, the functions of estimation and assurance within DND would be organizationally separated, to give further confidence that the cost assurance function is not only independent internally but also externally to the department; however, this may not be practical. The degree of organizational separation would depend on the ability of the organization to recruit and maintain a sufficiently large pool of costing experts to support both functions. However this is managed and practically implemented, it is important that these two functions are in place for the NGFC. The assurance role would typically be more involved at key milestones during the Development phase when decisions are being taken on the shape of the NGFC Program. The Assurance role might also be involved during other phases depending on the needs of decision makers and the agreed processes.

To successfully manage these two functions, formal policy and organizational arrangements are established to guide LCC Estimation and Assurance activities and an NGFC LCC plan would capture the planned LCC approach to LCC Estimation and Assurance for this project.

In comparison to JSF partner nations, we note that for Norway, F35 cost estimates are reviewed on an annual basis by an independent third party. Netherlands also performs an independent review on an annual basis through the Defence Audit Agency. In addition, the Netherlands Court of Audit publishes an annual independent report on the project. In Australia, the JSF project cost estimates are independently reviewed prior to submitting the annual update to Government. In the US, the Cost Analysis and Program Evaluation of the Office of Secretary of Defense undertakes independent cost estimates of all major projects and in the UK, the Cost Analysis and Assurances Services organisation located within the Ministry of Defence undertakes independent analysis of the cost estimates for major projects.

Organizational Procedures, Tools and Templates

Leading organizational practice also involves establishing standard procedures, tools and templates and ensuring the personnel are adequately trained and knowledgeable in these. This approach provides opportunities to transfer knowledge between work assignments as well ensuring the organization remain robust in dealing with personnel turnover and reviews.

To support ongoing work on NGFC costing activities, leading organizational practice supports that work is undertaken to establish, or gain access to, ongoing data sources to support future project analysis.

The LCC team composition for any assignment would depend on the complexity of the project, as well as any other sensitivities. Larger more complex projects submitted for Government approval for procurement funds with a substantive estimate would require a multidisciplinary team, as compared to early concept phase minor projects for which the preparation of indicative costs may only have required the efforts of one person.

4.3 NGFC LCC Principles: Processes

4.3.1 NGFC LCC Planning

This component involves the planning, by the NGFC LCC cost modeling team, to successfully achieve their task.

Leading Practices
  • The LCC development work is planned – an LCC Model/ Estimation Plan is developed
  • Project documentation, including technical, programmatic, acquisition strategies are defined
  • The LCC has a clearly articulated purpose
  • Plan to cost the scope of the project and not be influenced by the budget
  • Analysis of alternatives are described and a consistent approach used.
Current DND/TB Requirements

Treasury Board Policy states that Deputy Heads are responsible for ensuring that Departmental investment planning takes into account the whole-of-life cost of stewardship based on the life cycle of assets and acquired services20. We understand that the purpose of investment planning is to provide a better understanding of the affordability and strategic direction of planned investments. Treasury Board Policy also stipulates only a minimum period for the investment plans, noting that "the plan must clearly set out departmental priorities and strategies for the upcoming five-year period, and outline a three-year investment function that meets the needs of the department within available resources."

The DND Costing Handbook describes defining the costing requirement as a crucial factor in understanding what the costing is trying to achieve, and linking objectives to planned activities21.

Key NGFC LCC Framework Principles

As leading practice, the LCC plan is aligned to the purpose established in Section 5, and the nature of the NGFC Project is a key NGFC LCC Framework element. This plan is developed in consideration of the following:

  • Clear description of objectives of the LCC model and what outputs would be provided to support proposed management decision at each key stage
  • Clear description of the scope of the LCC in terms of both time frame and boundary in line with Section 4.3.2
  • A master schedule and resource requirement, including iterative development of the LCC as milestones are met and resources become available
  • Analysis of how data would be collected and normalized, and any limitations that are anticipated (by project phase and overall)
  • The Cost Breakdown Structure (CBS) and the costing method that would be applied to each element within the CBS
  • The means through which the LCC model would be independently reviewed ahead of each key milestone.

Types of alternatives that would be incorporated into the analysis, including at least:

  • A model that reflects the affordability within the authorized budget
  • Another model that meets the stated requirements, if the current requirements are not affordable
  • Whether, and how, analysis of alternatives/options would be managed.

It should be noted that the NGFC model would necessarily be an evergreen document that would continue to evolve throughout the program. The document is controlled through a formal configuration management approach, with clear understanding on who has authority to change the plan or elements of the plan.

Leading practice would suggest that the NGFC LCC be clearly focused on costing the endorsed capability, rather than costing to a budget or a subset of the endorsed capabilities.

The NGFC LCC is appropriately planned to achieve its established purpose to identify the LCC of the endorsed capability, and the plan considers the above noted elements.

If alternatives are being considered as part of the overall NGFC considerations, the alternatives are costed using a consistent framework, model, data and other aspects. Discounted cash flow techniques would be required to compare alternatives with differing cash flows.

4.3.2 Establish the boundaries/assumptions of the NGFC LCC Model

This element examines the need to establish clear boundaries for the NGFC LCC team to cost, as well as ensuring all key assumptions and supporting project documentation are fully documented and agreed.

Leading Practices
  • LCC estimate costing boundaries and assumptions are agreed with stakeholders
  • Project documentation, including technical, programmatic, and acquisition strategies are defined.
Current DND/TB Requirements

Prior to any TB submission approval, government departments are expected to initially communicate planned programs through an investment plan, which is required at least every three years. Plans must include, among other things, sufficient information on planned projects, including a minimum five year forecast, to support an informed decision by Treasury Board Ministers as to which projects would warrant their consideration over the planning horizon22. We consider it to be a floor requirement on the communication of boundaries and assumptions to TBS.

The DND Costing Handbook identifies the definition of objectives, linked to planned activities, as key components in determining the methodology to be used in preparing the estimate. Regarding assumptions, the DND Costing Handbook states that assumptions must be expressly stated and explained with adequate rationale to ensure that the results can be placed in the proper context. Within assumptions, it may be important to place boundaries in which certain cost elements may vary before rendering the costing results invalid. The DND Costing Handbook also states that "every cost analysis is unique and should not normally be used in another situation without ensuring that the circumstances and assumptions are the same."23 The DND Costing Handbook includes sections on Defining the Requirement, Categorizing Resource Cost, and Assessing Risk and Reasonableness, that each provide guidance on the definition and documentation of cost boundaries and assumptions.

Key NGFC LCC Framework Principles
Ground Rules and Assumptions

Because of information limitations (particularly early on), it is necessary to use a series of assumptions that constrain elements of the project in a meaningful way in order to allow the development of an LCC. These assumptions form a key element of the NGFC LCC as they define the basis on which the estimates are being developed. A separate document is maintained by the Cost Estimator that records all of these assumptions.

Ground rules can be thought of as the set of constraints that the Cost Estimator has been directed to use in the development of the estimate. The technical specifications can be thought of as such a ground rule. Assumptions are those that the estimator has used to develop in order to address any gaps between a projects given detail and ground rules and the total set of information necessary to develop an estimate.

This assumptions document should:

  • Identify global and program-specific assumptions, such as the estimates base year, including time-phasing and life cycle
  • Identify program schedule information by phase and program acquisition strategy
  • Identify any schedule or budget constraints, inflation assumptions
  • Specify equipment the government is to furnish as well as the use of existing facilities or new modification or development
  • Identify prime contractor and major subcontractors
  • Determine technology refresh cycles, technology assumptions, and new technology to be developed
  • Define commonality with legacy systems and assumed heritage savings
  • Describe effects of new ways of doing business
  • Describe the sustainment strategy as currently understood/assumed.

Where information does not exist, an assumption must be defined with the Project Management Office (PMO) to support the establishment of the best cost estimate. It is critical that the NGFC cost estimator has ready access to all available project documentation to support the development of assumptions and related cost estimates.

Cost Boundary

As outlined in Section 3.3 above, and within the DND Costing Handbook, to a large extent the applicable boundaries of the LCC model depend on the purpose of the LCC and the requirements of key stakeholders. For the NGFC Project team, the focus would be on the LCC elements that directly contribute to the NGFC. These costs are included in the following levels (as described in Section 3.3):

  • Contract Level
  • Project Level
  • Incremental Level
  • Program Level.

Comparison with other JSF countries identified that cost boundaries established generally align with the Program level:

  • Netherlands – All costs directly related to the F-35s are included procurement, in-service operations, and in-service support; including flying training, new infrastructure and use of current infrastructure. Industry support payments are not included.
  • Norway – the LCC includes all costs, including investment and all operating and sustainment costs over the lifetime of the aircraft.
  • Australia – all costs directly related to the JSF Project are included, MOU membership costs, acquisition, in service support and operations, infrastructure attributable directly to the JSF. Small amounts of support to industry are provided directly by the project.

4.3.3 Develop the structure of the NGFC LCC Model to complete the study

This element of the Framework involves those steps necessary to develop an LCC model and includes the following sub-steps:

  • Establish a CBS, based on the Work Breakdown Structure (WBS)
  • Create the LCC model
  • Identify the methods to be used to cost each element
  • Identify two independent methods for each cost element where possible
  • Ongoing documentation of the model.
4.3.3.1 Establish a Cost Breakdown Structure

The CBS provides a logical and complete breakdown of the NGFC Program such that if it is completely populated with accurate data the total LCC estimate would also be complete.

Leading Practices
  • LCC estimate uses standardized CBS tailored for the program under consideration, and consistently applied across all aspects of the project, including tendering
  • The CBS is derived from the WBS and provides a complete estimate of all costs within the designated boundaries of the program.
Current DND/TB Requirements

The DND Costing Handbook identifies a Work Breakdown Structure as an important part of the planning process, from which a Cost Element Structure (CES) can be developed. The guide considers the detail in the CES to be essential in developing the minimum level of detail for costing analysis24. We understand the CES to be the equivalent of a CBS.

Key NGFC LCC Framework Principles

As leading practice, the NGFC LCC makes use of a tailored CBS25 applied across all aspects of the program. A CBS includes a hierarchical decomposition of a project into manageable elements (i.e. subsystems, components, services and work packages) aligned to the Work Breakdown Structure but includes the use of cost categories and the notion of time. The WBS is a tree type structure that shows a subdivision of effort and defines the work required to achieve an overall end product related objective (such as, a Program System) and subordinate objectives, which provide for supporting or sub-projects and contracting arrangements. Products may include physical items or services which may be deliverable or enabling. The WBS defines and organizes the total scope of a project. The CBS may differ from the WBS in breaking down the costs to lower levels to support the cost estimation process. All JSF Partner countries responding to our request for consultation appear to use detailed CBS which include procurement and through life Sustainment and Operating costs.

An illustrative, high level CBS for the NGFC has been developed and provided in summary form in Figure 12, and in further detail in Annex B.

Figure 12: Cost Breakdown Structure Example
Cost Breakdown Structure Example. Text version below:
Figure 12 - Text version

This figure is a block diagram in four layers. The first layer is labelled "1.1 Defense (DND & CF)" which leads to the second layer, which is labelled "1.1.1 NGFC Capability" and "1.1.2 Other Defence Capability.", which leads to four blocks at the third level: "1.1.2.1 Personnel", "1.1.2.2 Equipment", "1.1.2.3 Infrastructure" and "1.1.2.4 Readiness".

The second level block "1.1.1 NGFC Capability" leads to six blocks at the third level: First, "1.1.1.1 Identification Stage", which leads to fourth level blocks titled "1.1.1.1.1 Management", "1.1.1.1.2 Studies, Analyses and Simulation" and "1.1.1.1.3 Other", second, "1.1.1.2 Options Analysis Stage", which leads to fourth level blocks titled "1.1.1.2.1 Management", "1.1.1.2.2 Studies, Analyses and Simulation" and "1.1.1.2.3 Other", third, "1.1.1.3 Definition Stage" which leads to fourth level blocks titled "1.1.1.3.1 Management", "1.1.1.3.2 Studies, Analyses and Simulation" and "1.1.1.3.3 Other", fourth, "1.1.1.4 Implementation Stage (and close out)", which leads to fourth level blocks titled "1.1.1.4.1 Management", "1.1.1.4.2 Studies, Analyses and Simulation", "1.1.1.4.3 Engineering", "1.1.1.4.4 Purchase off the shelf (NGFC aircraft system", "1.1.1.4.5 Investment", "1.1.1.4.6 Manufacturing", "1.1.1.4.7 Systems Integration", "1.1.1.4.8 System Level Test, Evaluation, Trials and Demonstrations", "1.1.1.4.9 Deployment", "1.1.1.4.10 Government Furnished", and "1.1.1.4.11 Other", fifth to "1.1.1.5 In-Service Phase" which leads to fourth level blocks titled "1.1.1.5.1 Operation", "1.1.1.5.2 Mission Support", "1.1.1.5.3 Maintenance", "1.1.1.5.4 Replenishment", "1.1.1.5.5 Continuous Training", "1.1.1.5.6 Packaging, Handling, Storage and Transportation", "1.1.1.5.7 Sustaining Support", "1.1.1.5.8 Restoration", and "1.1.1.5.9 Other". Sixth and last is the block titled "1.1.1.6 Disposal Phase" which leads to fourth level blocks titled "1.1.1.6.1 Management", "1.1.1.6.2 Studies, Analyses and Simulation" and "1.1.1.6.3 Other"

The second level block "1.1.2 Other Defence Capability leads to leads to four blocks at the third level: "1.1.2.1 Personnel", "1.1.2.2 Equipment", "1.1.2.3 Infrastructure" and "1.1.2.4 Readiness".

4.3.3.2 Build the LCC Model

The LCC Model is the tool that captures all inputs, undertakes the necessary calculations to provide outputs suitable for consideration.

Leading Practices
  • Leading practice modelling standards are used26 when developing and managing LCC tools/models
  • LCC model and estimate is subject to configuration control against the baseline cost model and estimate.
Current DND/TB Requirements

The TB Policy on Investment Planning requires that a departments investment plan take into account the whole-of-life cost of stewardship based on the life cycle of assets and acquired services27, but does not provide detail as to what that specifically would require. While it is not a mandatory policy, one of the key principals of the TB Guide to Costing is that "data and documentation must be reasonable, consistent, defensible, reconcilable, and current". The principal states that all assumptions, processes, and calculations used to produce the cost information, along with the data sources, are reviewed at least every two years to ensure continuing validity28. In addition, the DND Costing Handbook provides further guidance on the costing process (overview and detail).

Key NGFC LCC Framework Principles

As leading practice, the NGFC LCC model considers relevant model characteristics included in similar financially based models. In particular, the model itself should be29:

  • Accurate – Captures costs and Cost Estimating Relationships (CER) that are suitable to purpose and unbiased. Properly reflects uncertainty in data and agreed risks and risk treatments
  • Comprehensive – Matched to a Work Breakdown Structure that fully captures the program. All cost driving ground rules and assumptions are properly documented
  • Replicable and Auditable WBS elements are fully traceable to the system specifications. Estimate is thoroughly documented, including source data and significance and goodness of fit statistics for CERs that are also clearly documented and explained. From the information provided, a reviewer should be able to repeat all calculations and achieve the same results
  • Traceable – Data is traceable back to source documentation. WBS structure is aligned to the organizational structure conducting the work
  • Flexible – Estimating techniques should be allowed to vary as a program progresses through the various acquisition phases
  • Credible – Following the principles of the NGFC LCC Framework would support the delivery of a credible result. This can be further enhanced through the use of independent reviews as part of a standard quality control process
  • Timely – Results must be available in a manner that suits the timing of decision makers. The potential impact of insufficient time to conduct analysis must be quickly and clearly communicated to decision makers along with the potential limitations this could bring to the LCC estimate.

Other key principles with respect to the LCC Model include: fully documenting the model, and having a configuration management plan to help ensure the model remains consistent with the above characteristics and any changes are well considered and executed.

In terms of developing the model, there are specialized commercially available tools that support leading practice LCC estimation, model management and development. More generalized tools, such as Excel, require greater use of policies and procedures to implement leading practices and generally inclusion of additional third party applications (Excel Add-ins).

Comparison with other JSF Partner nations identified that:

  • Netherlands – LCC analysis and methodology follows FEL-SALDO methodology and is prepared using Excel30
  • Norway – LCC is prepared using an Excel based tool and then transferred to the risk analysis tool called Baseline to calculate LCC31
  • Australia – LCC is prepared using an Excel based standardised cost template though they are now in the process of investigating alternatives to this32.
  • US – the Cost Assessment and Program Evaluation (CAPE) office uses Excel in the development of their LCC estimates.
4.3.3.3 Identification of the methods to be used to cost each element

The baseline estimate for each cost element would be calculated based on the approach specified within the plan. Ultimately, each cost element would be aggregated to establish the LCC baseline estimate. Over the phases of the project, the methods used would evolve from more analogous and statistical approaches, to engineering, then to the use of actual data as illustrated in Figure 13, over leaf.

Leading Practices
  • Cost each element in accordance with the most appropriate cost estimating technique
  • Validate the LCC estimate with available historical data.
Current DND/TB Requirements

The DND Costing Handbook states that the approach must be logical and the costs must be reasonable, valid, relevant, and verifiable. The use of a specific approach varies with the amount and reliability of available data. Furthermore, more than one cost estimating method in the preparation of an estimate may be used33. The DND Costing Handbook also identifies four potential cost estimating techniques: engineering, parametric, analogy, and expert opinion.

Key NGFC LCC Framework Principles

As leading practice, the NGFC LCC costing team must identify for each costing element the methods they propose to apply in developing costs. For the early phases of the LCC estimates, more than one cost estimation technique is used (for example, parametric and analogous) to develop the costs for the most significant cost elements. This would give insight into the relative accuracy of the results. The approach taken, and cost estimation technique used, considers:

  • Feasibility of obtaining data
  • Newness of technology
  • Stage of the life cycle analysis
  • Other planned procedures.

Although there are numerous methods for LCC estimation available, the four most common methods are described below and the illustrative circumstances under which each approach might be used are illustrated in Figure 13:

  • Engineering cost method – Involves direct estimation of a particular cost element by examining products component by component. This approach is sometimes characterized as a 'Bottom up Approach
  • Analogous cost method – Involves cost estimation based on experience with the same or similar products or technology
  • Parametric cost method – Uses significant parameters and variables to develop cost estimating relationships
  • Extrapolation from actuals – Uses actual contract and project performance to estimate costs at completion including estimate of actual learning against projected, learning curves and earned value management approaches.
Figure 13: Depiction of cost method by project/program stage (adapted from SCEA CEBOK) 34
Depiction of cost method by project/program stage (adapted from SCEA CEBOK). Text version below:
Figure 13 - Text version

This figure relates different costing methods to project/program stages. The top of the diagram is a rectangle with the six project/program stages listed in order: Identification, Options Analysis, Definition, Implementation, Closeout, Operations & Sustainment and Disposal.

Underneath this rectangle is another rectangle segmented into four different areas that relate to costing methods. The area under the identification stage is mostly occupied by the Analogy segment, with some area occupied by the Parametric segment. The area under the Options analysis stage is mostly occupied by the parametric segment with some of the analogy segment. The area under the Definition stage is occupied by a small area of the Engineering segment, but is mostly occupied by the Parametric segment and the Analogy segment. The area under the Implementation and closeout stages are increasingly occupied by the Extrapolation from Actuals and Engineering segments, and rapidly decreasing parametric and analogy segments. The area under the Operations and Sustainment stage is mostly occupied by the Extrapolation and Actuals Segment and the Engineering segment, with only a small area occupied by the Parametric segment. Finally the area under the Disposal stage is mostly occupied by the extrapolation from Actuals, with some area occupied by the Engineering segment.

As leading practice, the NGFC LCC costing team must identify, for each costing element, the methods they propose to apply in developing costs. For the early phases of the LCC estimate, more than one cost estimation technique may be necessary (for example parametric and analogous) to develop the costs for significant cost elements.

4.3.4 Establish and document the data and populate the NGFC LCC Model

This component involves collecting data and incorporating it into the model, and establishing an initial point cost estimate or cost estimate range.

4.3.4.1 Data collection and normalization
Leading Practices
  • Data is collected and normalized
  • Appropriate investments are made to increase the accuracy and/or availability of key data elements
  • Cost elements that have a significant impact on the overall LCC estimate are identified.
Current DND/TB Requirements

The DND Costing Handbook references the importance of collecting accurate and detailed validated data, detailing the three specific steps as: Identify a source, collect the data, and validate the data to ensure it meets requirements and expectations. The DND Costing Handbook states that every effort should be made to ensure historical data is not distorted due to extraordinary conditions such as major exercise or equipment refit35. Section 10 of the DND Costing Handbook also provides illustrative examples of how and when to apply normalization techniques when preparing business case analysis.

Key NGFC LCC Framework Principles

Normalization is the process by which data is made consistent and comparable to other data being used in the estimate36. The most common example is the use of indexation to reflect earlier years price basis in todays dollars in order to compare. Under leading practice, data is collected and normalized in accordance to the plan – making any logical adjustments as necessary, reporting these to the project manager as required, and recording those changes through updates to the plan. Any data collected is appropriately normalized to account for issues such as:

  • Inflation
  • Foreign exchange
  • Activity levels
  • Differences in aircraft operations between the Canadian Air Force and US Air Force

The key objective of normalizing data is to help ensure that the data can be reasonably compared or used with respect to the LCC of the NGFC. It should also be clear from the above that the data itself is not just financial, but any data that provides insight and context to the NGFC LCC.

As a system moves through the various phases, data increases in quantity, reliability and nature (i.e. analogy, parametric or engineering). Also, consider section 3.5 for further explanation. Careful consideration should be given to help ensure cost data and its management continues to evolve. Other analysis that should be considered to assess if other costs need to be accounted for in the model include: cost/benefit analysis, trade-off studies, and support concepts.

As leading practice, all data is assessed for cost drivers or trends, and results compared to any standard factors, such as those produced through an economic model.

In addition, the Cost Estimator is expected to conduct interviews and assess data reliability, and consultations with other organizational sources should be undertaken.

4.3.4.2 Develop the Baseline Estimate

The baseline estimate is the estimated costs of the system without any provision made for risk.

Leading Practices
  • Cost each element in accordance with the most appropriate cost estimating technique
  • Cost the scope of the project, not to the budget
  • Validate the LCC estimate with available historical data and independent costing methods.
Current DND/TB Requirements

The DND Costing Handbook states that the approach must be logical and the costs must be reasonable, valid, relevant, and verifiable. The use of a specific approach varies with the amount and reliability of available data. Furthermore, more than one cost estimating method in the preparation of an estimate may be used37. As there are no mandatory policy requirements for the development of a baseline estimate, we have used the DND Costing Handbook to be the floor requirement to be considered.

Key NGFC LCC Framework Principles

Once the appropriate method has been applied, enter data against each cost element, expressing costs in the form that they are received, with the model being able to transform the data into the appropriate baseline within the model. Results are time phased in accordance with the schedule. It is important that the cost estimate is an unbiased representation of the expected costs of the systems, aligned to the agreed assumptions and other endorsed documentation. The cost analyst should take no account of the current budget or any other factors related to affordability or program approval criteria.

Once the model has been run, the Cost Estimator undertakes an examination of their own cost model using a standardized set of procedures and tools to ensure accuracy of results. At this time the Cost Estimator should identify and document key cost drivers within the model. These would be further examined in later stages of the process.

Documentation throughout the modelling process, including documenting the steps taken by the analyst to internally validate the model, is a leading practice.

4.3.5 Review, analyze and update the NGFC LCC results

4.3.5.1 Sensitivity Analysis

Sensitivity analysis is the process whereby key assumptions and ground rules in the LCC estimate are changed and impact on costs observed. In Sensitivity Analysis all other inputs are held constant, while a single variable is adjusted to observe the impact on the total cost estimate. This approach enables decision makers to be informed of cost sensitivities to varying, for example, system performance, schedule, quantities, basing strategies, support concepts, labour rate, etc.

As a program matures, the nature of the sensitivity analysis would change. Initially, the focus would be on the identification of the most sensitive cost drivers. Further analysis would be conducted on these drivers in order to support a deeper appreciation of how sensitive the costs are to these drivers, and to ensure an appropriate focus on improving knowledge.

Leading Practices
  • Conduct sensitivity analysis
  • Sensitivity analysis results are documented and communicated.
Current DND/TB Requirements

Risk management is considered a key consideration for Departmental investment planning as required by the TB Policy on Investment Planning – Assets and Acquired Services38, and has been considered a floor requirement in comparison to leading practice.

In addition, we note that a section of the DND Costing Handbook is dedicated to Risk Management and Uncertainty. The Handbook indicates that it is essential that the costing be assessed from the perspective of both risk and reasonableness to determine its validity. The DND Costing Handbook considers sensitivity analysis to be "identifying major sources of uncertainty, and to provide valuable information to the project team by highlighting elements that are cost sensitive, areas in which more research is needed to overcome cost obstacles to gain better project performance, and areas where system performance can be upgraded without substantially increasing project cost". In performing sensitivity analysis, the DND Costing Handbook suggests that all approaches, assumptions and models should be documented, as well as factors determined to warrant the sensitivity analysis39.

Key NGFC LCC Framework Principles

Sensitivity analysis is a very useful tool for aiding in the management of uncertainty and project cost risks. There is always a level of uncertainty and cost risk related to LCC estimates and key assumptions and this is especially so in the early stages of a projects life. Sensitivity analysis would give a range of possible high and low costs around a single point estimate and allow for better informed decisions to be made on which risk mitigation strategies would have the most impact.

The choice of which key assumptions and parameters to change and by how much, needs to be carefully considered and supportable as realistic alternatives to the baseline assumption or parameter.

Within the NGFC LCC the following variables could be considered:

  • Fluctuations in foreign exchange
  • United States currency/Canadian currency mix
  • Underpinning indices (i.e. inflationary, fuel, etc.)
  • Timing of aircraft procurements
  • Aircraft production volumes
  • Price volatility
  • Higher and lower learning curves
  • Fuel usage
  • Flying hours
  • Number of aircraft.

Part of this analysis includes an examination of whether the level of knowledge around any key cost drivers can be improved. Results of this examination are documented.

From the Sensitivity Analysis a short report can be presented that can quickly demonstrate the analysis. This might be portrayed using a Tornado graph such as Figure 14.

Figure 14: Tornado Graph: Sensitivity Analysis
Tornado Graph: Sensitivity Analysis. Text version below:
Figure 14 - Text version

This figure is a bar chart with five bars. The horizontal axis is labelled Costs, with the left half labelled lower costs and the right half labelled higher costs. The bars are labelled Element 1, Element 2, Element 3, Element 4 and Element 5. These bars are of differing lengths, where the length indicates the plausible change. The bars are situated over, to the left or to the right relative to the mid-point of the horizontal axis

4.3.5.2 Conduct Risk and Uncertainty Analysis and establish risk adjusted estimate.

Risk and uncertainty are related; uncertainty is the probability, while risk is the probability and consequence. Decision makers need to be informed of cost risks and uncertainties relevant to the cost estimates submitted for consideration. Quantitative risk and uncertainty analysis provides a way to assess the variability in the point estimate. Clear and concise statements need to be made about these cost risks and uncertainties, the likelihood (probability) of occurrence and the potential impact on the LCC estimate number. This analysis can model such effects as technical changes, schedules slipping, missions changing, and proposed solutions not meeting user needs, allowing for a known range of potential costs. Having a range of costs around a point estimate is more useful to decision makers, because it conveys the level of confidence in achieving the most likely cost.

Leading Practices
  • Risks and uncertainties are considered during the development of the LCC estimate and are identified where possible at the cost element breakdown level
  • Create a risk adjusted cost estimate (S Curve) as part of this aspect of the analysis (see Figure 15 overleaf)
  • Review and update results based on feedback.
Current DND/TB Requirements

Risk management is considered a key consideration for Departmental investment planning as stated in the TB Policy Framework for the Management of Risk40. This TB Policy Framework contains general statements that include the responsibility for Deputy Heads to ensure that risk management principles and practices are understood and integrated into the various activities of their organization. This is considered to be a floor requirement in comparison to leading practice.

In addition, we note that a section of the DND Costing Handbook is dedicated to Risk Management and Uncertainty. The DND Costing Handbook indicates that it is essential that the costing be assessed from the perspective of both risk and reasonableness to determine its validity. The DND Costing Handbook identifies the depiction of the estimate and its related uncertainty in the form of a cumulative probability distribution as one of the best methods to portray the uncertainty of an estimate. In determining the communication of the cost estimate, the DND Costing Handbook notes that "by establishing a confidence interval for the best estimate, there is recognition that the actual cost is likely to fall within a range of values. With that in mind, the cost analyst can best quantify cost risk by assigning a probability to possible outcomes and quantify risks if they occur"41.

As an example of confidence levels for data categories, Annex A of the DND Costing Handbook provides the following examples:

  • Rough order of Magnitude – <60% Confidence
  • Rough order of Magnitude/Indicative (based on rules of thumb or other analogous cost methods) – 60-70% Confidence
  • Indicative (based primarily on parametric cost methods) – 70-80% Confidence
  • Substantive/Indicative (based primarily on engineering cost methods) – 80-90% Confidence
  • Substantive (based primarily on costs extrapolated from actual and engineering cost methods – >90% Confidence

In the process of assessing risk, the DND Costing Handbook states that the use of a consistent Risk Assessment Methodology, "so that factors that are less apparent in the areas of technical and performance related risk can be examined for their potential influence"42.

The DNDs Project Approval Directive points out that for substantive43 cost estimates which are required at a much later stage for Expenditure Approval, that the contingency levels could be as high as 15% or even 20% depending on the totality of the project risk. One assumes that the contingency level of the NGFC which is pre Project Approval would be higher than the 15% to 20% range.

Key NGFC LCC Framework Principles

A projects Life Cycle Cost estimate is a forecast of a future cost based on project information known at the time that the estimates are developed. In the early stages of a projects life, project documentation is immature, studies to flesh out the requirements are still to be developed, and any cost data available is most likely based on analogous capabilities and/or the use of parametric modeling. There are therefore, in these early stages, significant uncertainties and cost risk surrounding the LCC estimate.

In order for the NGFC Project to implement leading practices on cost risk and uncertainty analysis, the following should be implemented in the development phase:

  • Risk management should be undertaken in accordance with recognized DND policy. For NGFC, this might include a Technical Readiness Assessment of Canadian-specific modifications.
  • A standard operating procedure is developed for the determination of the projects cost drivers and associated risks and should implement the procedure and record cost drivers and associated risks as a basis for determining uncertainty and overall cost risk.
  • Stakeholder engagement is sought for input into cost risk considerations (the what, the when and the potential individual cost element range). This engagement, where possible, should include other JSF partner nations as they would experience similar issues and may bring insight into how they go about handling this issue.
  • The NGFC Project Risk Log must indicate which cost risks and uncertainties have been subject to, or included, in the LCC.
  • Uncertainty analysis is undertaken so that the probability associated with achieving the LCC estimate can be determined. Uncertainty results from three possible sources: limited data, human unpredictability, and errors of observation or measurement.
  • An approach is to do statistical calculations to describe the variance associated with an analogy or a cost estimating relationship. The most common approach is to collect data on the optimistic, most likely, and pessimistic range (the "3-point estimate") for the cost element. The 3-point estimate and an idea about the distribution shape can be used to define the probability distribution to be used in a simulation.
  • A cumulative probability distribution, more commonly known as an S Curve (Figure 15), can show the uncertainty of various cost estimates. The S curve can be derived using statistical methodologies such as Monte Carlo simulation. In the simulation the lower level cost element probabilities are combined to give an overall project probability distribution. Monte Carlo simulation can be implemented quickly once all the data have been collected.
    Figure 15: S Curve
    S Curve. Text version below:
    Figure 15 - Text version

    This figure is a line graph where the x axis is labelled cost estimate (from zero to unknown dollars) and the y axis is labelled Percentage Likelihood not exceeding cost, ranging from zero to 100 percent. The line on the graph is an ‘S' curve meaning it starts at the origin, curves upward in an increasingly vertical curve, and then changes direction to a decreasingly vertical direction to return to the horizontal. The point at which the curve begins to decrease its vertical tendency is identified on the graph, and lines are dropped from this point to the horizontal axis (where the intersection is labelled Point estimate), and to the vertical axis (where the intersection is labelled% likelihood not to exceed).

    The area in the graph on the horizontal axis to the right of the line dropped to the point estimate$ is labelled "Financial Impact associated with higher levels of likelihood". Similarly, the area on the graph on the vertical axis above the line to the% likelihood not to exceed is labelled "Likelihood of costs exceeding point estimate".

  • The probability distributions are modeled to various types of uncertainty (for example, program, technical, organizational, program management including cost estimating and scheduling).
  • Account for correlation between cost elements to properly capture risk. As different CBS elements costs may be affected by the same external factors, some degree of correlation exists between them.
  • The NGFC Project Risk Log should indicate which cost risks and uncertainties have been subject to or included in the uncertainty analysis addressed above, and have adequacy of contingency allocation to mitigate risk.
  • Produce a risk adjusted estimate based on the above analysis, utilizing.to establish a sufficient level of confidence to meet Governments risk tolerance.
  • Depending on the stage of the project development, the risk adjusted estimate may form the initial budget, or be used to assess whether the current budget is appropriate or whether risk/cost reduction activities may need to be conducted. The NGFC already has an established budget.
Appropriate level of confidence for a risk adjusted estimate.

GAO states that "no specific confidence level is considered a best practice, experts agree that program cost estimates should be budgeted to at least the 50 percent confidence level, but budgeting to a higher level (for example,70 percent to 80 percent, or the mean) is now common practice."44 The GAO suggests that the mean of this distribution falls between 55 percent and 65 percent, with more risky projects at the higher end.

In his review of UK Defence Acquisitions45, Bernard Gray recommended funding at the 90 percent level to address the systematic under-forecasting during the early stages of a project. We understand that current practice in the UK is to fund projects between the 50% level and the recommended 90% level, or roughly the 70 percent level. JSF costs are presented by the JSF Program Office (JPO) in the SAR 11 report at the 50 percent level46. We also understand that NASA is required to fund projects at the 50 percent level, but the overall Program budget is to be set at the 70 percent level47. Norway has advised that they provide a project target cost at the 50% level as defined in the risk analyses, whilst the approved parliamentary budget is at 85 percent level (i.e. including risk contingency). The annual budgets are set at the annual P-50 level.

4.3.5.3 Document Results

This aspect of the documentation focuses on the documentation of the NGFC LCC for internal review (within DND).

Leading Practices
  • Document the estimate
  • Documentation of model should be a continuous process undertaken at every stage of the process.
Current DND/TB Requirements

The TB Guide to Costing indicates that Departments should document all assumptions, processes, and calculations used to produce the cost information48.

As previously noted in this document, the DND Costing Handbook suggests that the cost estimate process should be clearly documented throughout the process, including documentation of:

  • Assumptions and cost boundaries
  • WBS and CBS
  • Risk Assessment Methodology
  • Sensitivity analysis process49
Key NGFC LCC Framework Principles

Documentation is a key requirement of a cost model. Documentation is an ongoing activity to help ensure that key decisions or assumptions are not lost.

Documentation must have the following minimum criteria:

  1. Document the model such that another cost analyst unfamiliar with the program could recreate it quickly and produce the same result.
  2. Create an executive summary that provides sufficient explanation for a non-expert cost modeller to understand the costs and underlying assumptions.

Documentation should be routinely examined by other members of the LCC organization to help ensure that it remains appropriately updated and fit for task.

A key component of this work is to present the degree of uncertainty inherent in the estimate. The documented cost report includes the provision of an S Curve specifying the confidence level of the current baseline estimate and the current budget and the risk adjusted estimate. Other aspects of the report cover the detail necessary to present the report to outside stakeholders (external to DND).

4.3.5.4 LCC Assurance

As a standard component of the NGFC LCC, the model should be independently assured prior to any major milestone or in a manner consistent with the plan. This Assurance activity could be in the form of an independent review and/or the development of an ICE (as discussed in section 4.2.3). The primary purpose is to challenge the existing LCC estimate to ensure it is robust and reliable, taking into account the current life cycle of the project and knowledge of the system under investigation.

Leading Practices
  • An Independent Review of the LCC model/estimate is conducted.
Current DND/TB Requirements

The DND Costing Handbook states that all proposals must be put through a Cost Validation Process where an arms length review is performed to ensure that the following areas are reviewed:

  • Ensuring that all financial risks are known and visible
  • Understanding the financial implications of decisions before they are taken
  • Reporting on financial results
  • Protecting against fraud, financial negligence, violation of financial rules or principles, and losses of assets or public money50.
4.3.5.4.1 Key NGFC LCC Framework Principles

An ICE is recommended to be undertaken for the NGFC as part of the review process due to its complexity, size and strategic significance. The ICE process is commonly used for larger more complex projects within the United States Department of Defense and the United Kingdom Ministry of Defence.

The ICE considerations might include:

  • The cost estimating organization/group is independent of the originating cost estimates organization
  • The independent cost estimation organization needs to have skilled, experienced cost estimators with organizational back up support
  • Same access to data is made available to the independent cost estimator
  • The independent cost estimator is to follow the DND/TBS approved NGFC LCC approach
  • The independent cost estimator is to produce a standalone LCC Report which is suitable for the intended audience (usually senior management)
  • Once the ICE is complete, the Project costing team and Independent costing team document any differences and any factual errors (if any)
  • A final LCC Correlation Report is produced and approved by both the independent cost estimator and the PMO outlining the differences and the reasons for the differences between the two estimates
  • The NGFC LCC model should then be updated, if required, to accommodate any insights provided by the independent review.
  • The ICE is undertaken during the Definition Phase to support Project and Expenditure approval decisions.

Comparing JSF partner nations, we note that for Norway, F35 cost estimates are reviewed on an annual basis by an independent third party. The Netherlands also performs an independent review on an annual basis through the Defence Audit Agency. In addition, the Netherlands Court of audit publishes an annual independent report on the project. In the US, the CAPE undertakes an ICE for the JSF. In Australia, Defences Cost Analysis Branch undertakes an independent review of the JSF LCC prior to any submissions to Government and in the UK, Cost Analysis and Assurances Services organisation undertakes independent reviews of the JSF project.

4.4 Product Considerations

This aspect of the LCC Framework is focused on key decision makers and stakeholders who would, for the most part, have limited understanding of LCC principles or practices. The product considerations and resulting report(s) should be clearly focused on the purpose(s), which would also drive a number of critical assumptions in the model.

4.4.1 Interpret and report results

Leading Practices
  • An LCC Report is developed for management. This report allows management to clearly understand both the outcomes of the LCC estimate and the implications for the program.
  • A report should be focused on the purposes of the decision and decision maker and contain factual information relevant to that decision.
  • LCC data presented to decision makers for one decision is easily traceable to data presented to support earlier or other decisions. Where information has changed, the changes are clearly explained and supported by a robust audit trail.
Current DND/TB Requirements

The TBS Guide to Costing indicates that the purpose of the costing report is to produce information that supports the actions of managers and decision makers, assists TBS in assessing business cases, and enhances the reporting and public disclosure of information. Managers may also use the cost information to develop, for example, cost-benefit analyses, risk assessments, or cost recovery strategies.

As noted in section 4.3.5.2 of this report, Annex A of the DND Costing Handbook provides data categorization examples that would appear to suggest a confidence level to associate with cost estimates in their various stages of development. The examples would appear to indicate an expectation that once estimates have evolved beyond a Rough order of Magnitude, confidence levels would be above 60% during development, reaching 90% as the estimate becomes substantive51. The DND Costing Handbook notes that the Directorate of Strategic Financing and Costing works with a range of values, as opposed to point estimates.

Key NGFC LCC Framework Principles
Report Structure

Leading practice includes the use of a standard LCC analysis report structure which would bring out the key issues related to the costs presented in a concise, factual and easily understood manner. It includes details pertinent to the decision at hand, including the cost confidence level, risks and uncertainties, summaries of analysis such as sensitivity, risk and affordability analysis, recommendations and conclusions. The report does not assume the reader has a detailed understanding of LCC principles and careful attention should be paid to the expression of uncertainty.

Reporting Level of Confidence (see also Section 3.5)

The key focus of interest of senior decision makers from the LCC would be the actual cost, and the level of uncertainty that surrounds those costs. Both these elements would be significantly impacted by the purposes for which the decision maker has for using the information.

As a general guide, decisions that are focused on budgets or funding approval would be most interested in the level of costs that they have direct authority over, and the time frame that is consistent with their budgetary responsibilities.

In order to establish a reference for consideration of the Baseline Estimate and the Project Budget (Baseline + Contingency) it is recommended that these costs be presented against a range of confidence levels that would provide reference points for the decision maker.

The NGFC has an approved budget. We believe that what is of most interest to decision makers is affordability. If the current project budget falls below the 50 percent confidence level the project should be considered as having a high risk that it is inadequately funded. Based on the above analysis, and in consideration of the US, UK and Norwegian practices, a decision maker will draw increasing comfort from a budget at least at the 50 percent level or above.

If desired, the decision maker could establish a prescribed confidence level that the project should provide options to achieve. For example, they might prescribe a confidence level of 65 percent to reflect the overall conditions of the project at this stage in the cycle. In this case, DND would be required to bring two business cases. The first case would present an LCC that reflected the currently agreed requirements and another case that reflected the currently approved budget based on the prescribed confidence level. A visual way of presenting this data is provided in Figure 16, overleaf.

Report Period

The time frame for the LCC analysis is aligned to the purposes of the decision maker. The life cycle cost time frame is from Development to Disposal. However, short periods are appropriate where the decision maker, or stakeholder, do not require such information. Consequently, the time frame for presentation may vary between uses, but each LCC analysis is drawn from and reconcilable to a single consistent source of information.

Summary

In summary, leading practice indicates that at a minimum, cost estimates are prepared at a 50% confidence level. Leading practice also suggests that funding submissions and budget planning be prepared at a higher confidence level, likely in the 70-85% confidence range. Selection of the appropriate confidence levels and reporting period for the various project phases should be based on management judgment, in consideration of factors including the purpose for which the cost estimate is to be used, quality of data available, and other relevant factors.

Figure 16: Presentation of LCC uncertainty
Presentation of Life Cycle Costing uncertainty. Text version below:
Figure 16 - Text version

This figure, labelled "Presentation of LCC uncertainty" shows a horizontal axis with three anchors: Low estimate (10% confidence), Mid-Point (50% confidence) and High estimate (90% confidence). Between the low and mid-point is another anchor labelled "Baseline Estimate at x% confidence". Between the mid-point and the high estimate is a fifth anchor labelled "Approved Budget at x% confidence". Sitting above the High Estimate anchor is another anchor labelled "Prescribed Confidence Level at X % and Y million dollars".

Below the line are three levels of blocks. Centred below the mid point is a block titled "Available contingency $x million = x%

There are two blocks at the second level that are followed by a two-ended arrow that runs the length of the axis. The first block, occupying the area from the low to mid point on the axis is a block titled "Key strategies to manage risks and reduce/contain costs". Below the mid point to the high point on the axis is another block labelled "Key risks that could lead to higher costs".

Below the two-ended arrow is another block that also spans the length of the axis. It is titled "Key assumptions that underpin estimate that have not been included in risk/uncertainty analysis"

5 Considerations in Implementing the NGFC LCC Framework Principles

5.1 Importance of Judgment

Within the federal government, Ministers and deputy heads have authority to manage the people, resources and activities of their departments towards the objectives set out in legislative mandates and government policy. The Treasury Board of Canada Secretariat, as the management board, has the authority to ensure that the government as a whole is managed in a coherent and effective manner. The government's management regime establishes minimum standards for how Ministers and deputy heads use their authorities and manage public resources. This management regime consists of sound management practices, strong public service values, and clear rules set out in legislation and Treasury Board policies.52

The NGFC LCC Framework is principles-based, and therefore requires the judgment referenced above if it is to be used to support the NGFC LCC program. These judgments are guided by the principles, but ultimately be based on producing the most relevant and accurate LCC estimate applicable for the intended purpose to meet stakeholder needs.

To illustrate this point, four topical examples are presented below:

5.1.1 Attrition Aircraft

If the JSF is acquired by the Canadian government, we understand attrition of aircraft is anticipated (i.e. beyond economic repair) for up to 14 aircraft53 over the life of the capability, based on the initial aircraft number of 65 (please note that we have not substantiated this number further than what was contained in the OAG Report). Without replacement, over time the capacity of the DND to generate operational capability from the JSF would naturally decline. That is, if 65 aircraft are the minimum number to meet the requirements of Government, then over time DND would either no longer be able to meet these requirements, or would only be able to meet them with increased risk. This is presented in a stylized manner below.

Figure 17: Aircraft Attrition Risk
Aircraft Attrition Risk. Text version below:
Figure 17 - Text version

This is a line graph with the horizontal axis labelled Time and the vertical axis labelled NGFC aircraft. The line of the graph, labelled "Available aircraft", starts at the top of the vertical axis with a label of 65 aircraft, runs down to the right and ends above the end of the horizontal axis with a label of 51 aircraft.

The area below this line is commented "the less aircraft in the fleet the greater the risk an aircraft will not be available to perform an essential mission".

From the point identified on the vertical axis as 65 aircraft, there is a horizontal line (parallel to the horizontal axis). This line is labelled "Reported minimum required number of aircraft".

The area between the line labelled "Reported minimum required number of aircraft" and the line depicting the number aircraft of is triangular in shape, shaded and labelled "increasing gap between required and available aircraft". The gap between the label of 51 aircraft and the horizontal line labelled "Reported minimum required number of aircraft" is labelled "Aircraft lost (attrition)"

Should, therefore, the costs of attrition aircraft be included in the LCC estimate? The guiding LCC Framework principles are that the LCC estimate should reflect the requirements (or scope) of the project and program, and that the detailed requirement relating to number of aircraft at various points in time in the future. While past practices for attrition are a relevant consideration, current practice should be based on the purpose of the LCC estimate being planned.

If the requirement is to maintain a certain capability over a period of time, a prospective estimated loss of capability of 14 aircraft may not be deemed acceptable and as a result, the LCC estimate may consider including attrition aircraft.

Also, the parameters within which these life-cycle costs are included may also vary depending on cost hierarchy presented. For example, attrition aircraft may not be considered part of the current project cost (depending on the approach being taken), but may appear as a future cost at the program level.

At a minimum, the framework principle of transparency would suggest that a clear rationale for the attrition assumption and risk would be disclosed within the NGFC LCC reporting, and the assumption consistently applied within the LCC estimate.

Consultation information received from other JSF Partner nations identified the following aircraft attrition assumptions:

  • The Netherlands – have not stated specifically how attrition aircraft are costed and are still to decide on total number of aircraft to be procured. The intent was on the procurement of 85 but the final decision is to be made54.
  • Australia – Attrition aircraft are built into their baseline LCC estimate55.

5.1.2 Costs of future upgrades

Another question might be whether the costs of future upgrades should be included within the LCC estimate. Again, judgment, based on the LCC Framework principles, is fundamental to answering this question. The LCC Framework would suggest that any mandatory upgrades, "block upgrades" to maintain the NGFC capability would be included within the Program level sustainment costs, based on the best information available. Upgrades required to maintain the relative capabilities of the aircraft would be included as these are part of the LCC of the capability. Future upgrades that would significantly alter the capabilities of the aircraft would likely not be considered as part of the LCC of the current capability requirements. Judgments would be necessary to determine whether the upgrades are required to maintain Canadas NGFC aircraft capability, or fundamentally enhance the capability. Assumptions with respect to these upgrades should be informed by JPO information, considerations related to expectations out of JSF partners, as well as an analysis of historic air programs, such as the CF-18.

At a minimum, the framework principle of transparency would suggest that a clear rationale for the future upgrade assumptions would be disclosed within the NGFC LCC report, and consistently applied within the LCC estimate.

5.1.3 Lifespan to be used for LCC reports

A key question is what timeframe should be used for LCC reporting. The LCC Framework principle would suggest that the LCC would include development, acquisition, support & operations and disposal costs - that is all actual and estimated costs over the program life (project initiation through to the disposal of the aircraft). This is further supported by the definition of a LCCE (Section 3.1) which describes the inclusion of all program cost elements "from concept all the way through operations, support, and disposal".

However, as was outlined in Section 3.3.2, shorter timeframes (and subsets of the cost elements) would be extracted from the LCC Framework for specific reporting purposes or decision points, and/or if reasonable assumptions cannot be reasonably substantiated to the span of years until disposal. For the specific purpose of providing an Annual Update to Parliament, identified in Section 3.3.2.4, the LCC Framework identifies that, to promote transparency and accountability, "the span of years for the project reporting should include all estimated costs out to the disposal of the NGFC, as well as already incurred costs.

The estimate of the useful life of the NGFC should be informed by the best information available at the time and documented assumptions and analysis and should consider information received from the JPO. Consultation information received from other JSF Partner nations identified the following LCC lifespan assumptions:

  • Norway – "30 years after last delivery (currently 2025-205456)"
  • The Netherlands – "30 years57"
  • Australia – 30 years58

At a minimum, the framework principle of transparency would suggest that a clear rationale for the LCC lifespan should be disclosed within the NGFC LCC report, and consistently applied within the LCC estimate.

5.1.4 Initial NGFC LCC Approach

Cost Data:

The NGFC is yet to receive Project Approval and, as a result, elements of the Options Analysis phase are still being undertaken that would inform the development of project requirements and other project documentation, the cost estimates and the resulting Project Brief and submission for Project Approval and the related initial Expenditure Approval. As a result of being in the Development phase, with little formal project definition, broad assumptions would need to be made about other cost elements that would need to be included in the LCC. The cost estimates can only be as good as the state of maturity of the project definition documentation. Section 4.3.3.3 summarizes key considerations and principles for the estimating of the cost of each cost element. Feedback from the Netherlands identified that the "main source of information is from the JSF PO (JPO)" and they "use the most conservative estimate by either the JPO or CAPE59". "Until we will have a better view on how in service support will be organized in detail, we will use the information provided by JSF PO (JPO)."

Cost Element:

Another key consideration in the development of the NGFC LCC is what level of cost should be included in the LCC. As the LCCE (Section 3.1) defines, "...all cost elements that pertain to the program... (including) all past (or sunk), present, and future costs for every aspect of the program, regardless of funding source." In addition, Section 3.3 summarizes the various levels of cost hierarchy in consideration of the "requirements of key stakeholders". Section 3.3.2.4 further suggests that to in consideration of these principles and to promote transparency and accountability, the Annual Update to Parliament should "include all costs out to and including the program level."

In comparison, the following was reported from other JSF Partner nations, with respect to the boundaries of cost included in the LCC:

  • Norway – "The boundaries are defined in the WBS documentation.60"
  • The Netherlands – "all costs that are directly related to the F-35 are included... (including) costs for basic flying training.....building new infrastructure and using infrastructure", "however, industry support payments are not61."
  • Australia – acquisition of aircraft, "some industry support, weapons war stocks (Phase 3 and 5 of project), Facilities, changes in workforce, software support, IT infrastructure" and personnel and operating costs (including support).62
Confidence Level:

As outlined in Sections 3.4, 3.5 and 4.4.1, life cycle costing is fundamentally a forecasting activity and, as a result, there will always be a level of uncertainty within the cost estimates. Over the life of the project, the level of uncertainties in the costing will diminish. Key questions might be what level of certainty should be expected at each phase of the LCC and if/how this confidence level should be reported.

Because the NGFC is yet to receive Project Approval, elements of the Options Analysis phase are still being undertaken and certain cost elements remain at a ROM. It is also noted, however, that for significant elements of the program there is more cost information available to support the NGFC LCC than is typically available at this phase of a project. There is greater definition and cost details of the prime equipment, that is the F-35 aircraft, than normally expected at this stage. The DND Costing Handbook would suggest that ROM data is produced at a confidence level of less than 60 percent and further suggests that a "point estimate, in its usual form, will not suffice, as it does not reflect any adjustments for the uncertainty or risk being faced by the project63". As suggested in Section 4.4.1, considering the nature and phase of the project, availability of cost information, and purpose of the NGFC LCC to be presented in the annual update to Parliament, the NGFC project budget (baseline cost estimate plus contingency), should be established at a confidence level of a minimum of a 50 percent level or higher. As suggested above, confidence levels can be estimated and applied to a point estimate and are based on various attributes of the LCC estimate, and the additional contingency applied. Ranges of estimates can also be presented and/or other point estimates, with various levels of confidence to further support decision making (see Norway example below).

From our analysis of other JSF Partner nations (where responses were received and/or information was publically available), the following use confidence levels to support analysis and decision making with the JSF project at a similar stage to that in Canada or earlier:

  • United States – GOA states that no specific confidence level is considered a best practice, however experts agree that program cost estimates should be budgeted to at least the 50 percent cost level, but budgeting to a higher level (for example, 70 to 80 percent) is now common practice64
  • United Kingdom – fund projects at the 70 percent confidence level
  • The Netherlands– no directives exist for applying confidence levels, however, NATO RTO SAS-054 is used as a guide for how uncertainty analysis should be calculated
  • Norway – use 50 percent confidence level to establish the target cost and project budget at the 85 percent level65. The annual budgets are set at the P-50 level.

The following Nations do not apply confidence levels at this stage of the project:

  • Australia
  • Canada
Reporting Level of Confidence:

With respect to the reporting of the level of confidence in LCC reports, from our analysis of other JSF Partner nations (where responses were received and/or information was publically available), the following summarizes the reporting of confidence levels:

  • Norway – reported level of confidence is 50 percent for the target cost and 85 percent for the budget66. Furthermore for ROM estimates 30% of the point estimate is added to establish the "worst case" estimate and 30% removed to establish the point estimate for the "best case" scenario
  • Netherlands – "uncertainty analysis are not part of the information" provided to parliament67
  • Australia – while not using confidence levels in reporting, Australia does report a range of cost estimates in the public version of the Defence Capability Plan 2011.68

It should be noted that responses were not received by all JSF Partner nations. Also, the DND Costing Handbook identifies the importance of communicating cost risk to "ensure that decision makers are aware specifically of what the estimate includes or does not include".

Contingency:

Section 4.3.5.2 suggests that cost risks and uncertainties should be documented and related contingency established to mitigate this risk. Section 4.4.1 further suggests that the contingency established for budget purpose should be derived in consideration of the level of cost certainty (confidence level) desired for the estimate. Section 8 of the DND Costing Handbook also discusses the practice of adding contingency to a project budget to deal with uncertainty and increase confidence levels.

Our analysis of other JSF Partner nations regarding the methodology for calculating, and the respective level of contingency, identified the following:

  • Norway – "consider the contingency in every field (of the WBS) separately69"
  • Netherlands – "5% of the total budget amount except for project with high associated risk where a 10% contingency amount is applied." Given the current stage of procurement, "no contingency amount has been determined70."
  • Australia – contingency developed at cost breakdown structure element level and then rolled up. The JSF aircraft unit price from JPO was adjusted with additional percentage amount built into the base price and then contingency added on top of this.71

At a minimum, the framework principle of transparency would suggest that a clear rationale for the estimation of contingency should be disclosed within the NGFC LCC report, and consistently applied within the LCC Framework.

6 Summary of Key Principles

Figure 18: Summary of Key Principles
Summary of Key Principles. Text version below:
Figure 18 - Text version

This figure is a block chart that combines the aspects of Figure 11: LCC Framework Components a second, parallel column of blocks.

The block labelled "NGFC LCC Planning" is situated beside a block labelled "Develop plan to ensure NGFC LCC model meets needs of all prospective users and aligns with LCC Framework"

The second LCC framework block, labelled "Boundaries and Assumptions" is situated beside a block labelled "Well defined and agreed boundaries are established. Key ground rules and assumptions understood and agreed. Project documentation is readily available and forms the basis for costs"

The third LCC framework block labelled "Develop Model" is situated beside a block labelled "Cost Breakdown Structure (CBS) is developed representing total Program. Model developed in line with better practice and supporting CBS and the range of decision anticipated. Appropriate cost estimation method selected for each cost element."

The fourth LCC framework block "Data, Populate and Document model" is situated beside a block labelled "Collect and normalize data set. Develop the baseline estimate and conduct internal validation of model and results."

The fifth LCC framework block titled "Review, Analyse and update" is situated beside a block labelled "Undertake sensitivity, risk and uncertainty analysis and develop risk adjusted cost estimate. Establish and document results. Undertake independent cost assurance activities and make any necessary adjustments to NGFC LCC model."

The sixth LCC framework block titled "Interpret and report results" is situated beside a block labelled "Purpose focussed reports are developed for decision makers and stakeholders in accordance with prescribed guidelines."

These blocks are all underlined by a final block labelled "NGFC estimator team is drawn from a professional costing organization, supported by standard tools, techniques and methods. The Cost Assurance role is integrated into the process with appropriate policies to ensure a non-advocacy approach."

Annexes

Annex A - Key References

In developing the NGFC LCC Framework, key considerations and references included documentation and interviews to establish the minimum standards of practice in Canada and military specific leading life cycle cost methods and leading general life cycle cost methods.

Documents

The following documents or sources were considered in the development of these methods:

  • Aerospace Information Report SAE AIR5416 Maintenance of Life Cycle Cost Model
  • Aerospace Recommended Practice SAE ARP4293 LCC Techniques and Applications 1992
  • ANAO Audit Report on F35 Sep 2012
  • ANAO LCC Better Practice
  • AS/NZS 4356:1999 LCC An Application Guide
  • Best Practice Spreadsheet Modeling Standards Version 6.1 2010
  • Australia – AS IEC 60300.3.3-2005 Dependability Management Part 3.3 Application Guide – Life Cycle Costing
  • Australia – Department of Defence – Capability Plan 2012
  • Australian Defence Force DI (G) LOG 03-4 Defence Policy On Life Cycle Costing Analysis
  • Canada – A Guide to Preparing Treasury Board Submissions,
  • Canada – DND – Canada First Defence Strategy (PDF document – 5.18 MB)
  • Canada – DND Costing Handbook (Directorate of Finance and Costing),
  • Canada – DND Project Approval Directive 2011-2012
  • Canada – Report of the Auditor General of Canada – Spring 2012, Replacing Canadas Fighter Jets
  • Canada – Treasury Board Foundation Framework for Treasury Board Policies,
  • Canada – Treasury Board Policy Framework for the Management of Risk,
  • Canada – Treasury Board Guide to Costing,
  • Canada – Treasury Board Policy on Investment Planning – Assets and Acquired Services,
  • Canada – Treasury Board Policy on the Management of Projects,
  • DTM-09-027 Implementation of the Weapons Systems Acquisition Reforms Act of 2009
  • GAO Cost Estimation and Assessment Guide
  • GAO DoD F35 Program – Actions Needed to Further Enforce Restructuring and Address Affordability and Risks
  • GAO Tactical Aircraft F22A Modernization program Faces Cost, Technical, and Sustainment Risks
  • ISO 15686-5 Buildings and Constructed Assets Service Life Planning Part 5 Life-cycle Costing
  • NASA Cost Estimating Handbook 2008
  • NATO Code of Practice for Life Cycle Costing TR-SAS-069
  • NATO Independent Cost Estimating and the Role of Life Cycle Cost Analysis in Managing the Defence Enterprise
  • NATO RTO TECH REP TR-SAS-054 Methods and Models for Life Cycle Costing
  • NATO TR 58 Cost Structures and Life Cycle Costs for Military Systems
  • NSW Treasury Total Asset Management
  • RAND Technical Report – A Methodology for Comparing Costs and Benefits of Management Alternatives for F-22 Sustainment 2011
  • Society Cost Estimating and Analysis Cost Estimation Body of Knowledge (CEBoK)
  • UK DERA Report on Recent Combat Aircraft Life Cycle Costing 1999
  • UK MoD Acquisition Operating Framework 2012
  • UK MoD Forecasting Guide
  • UK MoD Generic Work Breakdown Structure
  • UK Review of Acquisition The Gray Report 2009
  • UK Treasury Green Book – Appraisal and Evaluation in Central Government
  • UK Treasury Orange Book – Risk

Consultations

In developing the NGFC LCC Framework, we have consulted the identified stakeholders and key departments to assist in the identification of the framework purpose and requirements.

  • Department of National Defense
  • Chief Financial Officer
  • Costing Services
  • NGFC Project Management Office
  • Treasury Board Secretariat of Canada
  • Costing Strategies
  • International Affairs, Immigration and Defense
  • International Affairs, Security and Justice
  • Investment Planning and Project Management
  • Office of the Comptroller General (Project Authority)
  • Finance Canada
  • Defence and Crown Corporations
  • Reserves and Risk Management
  • Public Works and Government Services Canada
  • Acquisitions Branch, Contract Cost Analysis
  • Acquisitions Branch, Major Projects Directorate
  • National Fighter Procurement Secretariat
  • JSF Partners
  • Australia
  • Canada
  • Netherlands
  • Norway

Annex B - Program Cost Breakdown Structure (Sample Only)

Cost Breakdown Sub-Structure – Implementation Stage (1.1.1.4)
CBS# Level 5 Level 6 Level 7 (provided for understanding scope for this group)
1.1.1.4.1.1 Management Project Management N/A
1.1.1.4.1.1.1 Management Project Management Accommodation
1.1.1.4.1.1.2 Management Project Management DMPS Support
1.1.1.4.1.1.3 Management Project Management Foreign Service Allowance
1.1.1.4.1.1.4 Management Project Management Personnel (Civil, Military, Reserves)
1.1.1.4.1.1.5 Management Project Management Personnel Cost Moves
1.1.1.4.1.1.6 Management Project Management PMO Miscellaneous (Office Items)
1.1.1.4.1.1.7 Management Project Management PMSO (Levy on MCPs)
1.1.1.4.1.1.8 Management Project Management Professional Services
1.1.1.4.1.1.9 Management Project Management Revenue Dependency (PWGSC Personnel & Studies)
1.1.1.4.1.1.10 Management Project Management Temporary Duty (PMO)
1.1.1.4.1.1.11 Management Project Management Training (Project Unique)
1.1.1.4.1.1.12 Management Project Management Translation (Services)
1.1.1.4.1.2 Management Systems Engineering N/A
1.1.1.4.1.3 Management Integrated Logistics Support N/A
1.1.1.4.1.4 Management Other N/A
1.1.1.4.2 Studies, Analyses & Simulation N/A N/A
1.1.1.4.2.1 Studies, Analyses & Simulation Definition Studies N/A
1.1.1.4.2.2 Studies, Analyses & Simulation Other N/A
1.1.1.4.3 NGFC Aircraft System N/A N/A
1.1.1.4.3.1 NGFC Aircraft System Airframe N/A
1.1.1.4.3.1.1 NGFC Aircraft System Airframe Airframe Integration, Assembly, Test & Checkout
1.1.1.4.3.1.2 NGFC Aircraft System Airframe Fuselage
1.1.1.4.3.1.3 NGFC Aircraft System Airframe Wing
1.1.1.4.3.1.4 NGFC Aircraft System Airframe Empennage
1.1.1.4.3.1.5 NGFC Aircraft System Airframe Nacelle
1.1.1.4.3.1.6 NGFC Aircraft System Airframe Other Airframe Subsystems
1.1.1.4.3.2 NGFC Aircraft System Propulsion System N/A
1.1.1.4.3.2.1 NGFC Aircraft System Propulsion System Engine Integration, Assembly, Test & Checkout
1.1.1.4.3.2.2 NGFC Aircraft System Propulsion System Engine
1.1.1.4.3.2.3 NGFC Aircraft System Propulsion System Thrust Devices
1.1.1.4.3.2.4 NGFC Aircraft System Propulsion System Afterburner / Boost Units
1.1.1.4.3.2.5 NGFC Aircraft System Propulsion System Engine Monitoring & Control
1.1.1.4.3.2.6 NGFC Aircraft System Propulsion System Engine Software
1.1.1.4.3.2.7 NGFC Aircraft System Propulsion System Other Propulsion Subsystems
1.1.1.4.3.3 NGFC Aircraft System Vehicle Systems N/A
1.1.1.4.3.3.1 NGFC Aircraft System Vehicle Systems Vehicle Integration, Assembly, Test & Checkout
1.1.1.4.3.3.2 NGFC Aircraft System Vehicle Systems Flight Control SS
1.1.1.4.3.3.3 NGFC Aircraft System Vehicle Systems Auxiliary Power SS
1.1.1.4.3.3.4 NGFC Aircraft System Vehicle Systems Hydraulic SS
1.1.1.4.3.3.5 NGFC Aircraft System Vehicle Systems Electrical SS
1.1.1.4.3.3.6 NGFC Aircraft System Vehicle Systems Crew Station SS
1.1.1.4.3.3.7 NGFC Aircraft System Vehicle Systems Environmental Control SS
1.1.1.4.3.3.8 NGFC Aircraft System Vehicle Systems Fuel SS
1.1.1.4.3.3.9 NGFC Aircraft System Vehicle Systems Landing Gear SS
1.1.1.4.3.3.10 NGFC Aircraft System Vehicle Systems Vehicle SS Software
1.1.1.4.3.3.11 NGFC Aircraft System Vehicle Systems Other Vehicle Subsystem 1
1.1.1.4.3.4 NGFC Aircraft System Mission Systems N/A
1.1.1.4.3.4.1 NGFC Aircraft System Mission Systems Missions Systems Integration, Assembly, Test & Checkout
1.1.1.4.3.4.2 NGFC Aircraft System Mission Systems Communications / Identification
1.1.1.4.3.4.3 NGFC Aircraft System Mission Systems Navigation / Guidance
1.1.1.4.3.4.4 NGFC Aircraft System Mission Systems Mission Computer / Processing
1.1.1.4.3.4.5 NGFC Aircraft System Mission Systems Fire Control
1.1.1.4.3.4.6 NGFC Aircraft System Mission Systems Data Display & Controls
1.1.1.4.3.4.7 NGFC Aircraft System Mission Systems Survivability
1.1.1.4.3.4.8 NGFC Aircraft System Mission Systems Reconnaissance
1.1.1.4.3.4.9 NGFC Aircraft System Mission Systems Automatic Flight Control
1.1.1.4.3.4.10 NGFC Aircraft System Mission Systems Health Monitoring System
1.1.1.4.3.4.11 NGFC Aircraft System Mission Systems Stores Management
1.1.1.4.3.4.12 NGFC Aircraft System Mission Systems Weapons Delivery
1.1.1.4.3.4.13 NGFC Aircraft System Mission Systems Mission Systems Software
1.1.1.4.3.4.14 NGFC Aircraft System Mission Systems Other Mission Subsystems 1
1.1.1.4.3.5 NGFC Aircraft System Ancillary Equipment N/A
1.1.1.4.3.5.1 NGFC Aircraft System Ancillary Equipment External Fuel Tanks
1.1.1.4.3.5.2 NGFC Aircraft System Ancillary Equipment Other Ancillary Equipment
1.1.1.4.3.6 NGFC Aircraft System Aircraft Software N/A
1.1.1.4.3.7 NGFC Aircraft System Engineering Changes N/A
1.1.1.4.3.8 NGFC Aircraft System Aircraft Integration, Assembly, Test & Checkout N/A
1.1.1.4.4 NGFC Support System (Sustainment Set-up) N/A N/A
1.1.1.4.4.1 NGFC Support System (Sustainment Set-up) Maintenance Support N/A
1.1.1.4.4.1.1 NGFC Support System (Sustainment Set-up) Maintenance Support Programming Lab
1.1.1.4.4.1.2 NGFC Support System (Sustainment Set-up) Maintenance Support ALIS
1.1.1.4.4.1.3 NGFC Support System (Sustainment Set-up) Maintenance Support Depot Stand Up
1.1.1.4.4.1.4 NGFC Support System (Sustainment Set-up) Maintenance Support Other Maintenance
1.1.1.4.4.2 NGFC Support System (Sustainment Set-up) Supply Support N/A
1.1.1.4.4.2.1 NGFC Support System (Sustainment Set-up) Supply Support Initial Spares
1.1.1.4.4.2.2 NGFC Support System (Sustainment Set-up) Supply Support Initial Repair Parts
1.1.1.4.4.2.3 NGFC Support System (Sustainment Set-up) Supply Support Initial Consumables
1.1.1.4.4.2.4 NGFC Support System (Sustainment Set-up) Supply Support Initial Weapons
1.1.1.4.4.2.5 NGFC Support System (Sustainment Set-up) Supply Support Reserve Weapons
1.1.1.4.4.2.6 NGFC Support System (Sustainment Set-up) Supply Support GSM
1.1.1.4.4.2.7 NGFC Support System (Sustainment Set-up) Supply Support Other Supply
1.1.1.4.4.3 NGFC Support System (Sustainment Set-up) Personnel N/A
1.1.1.4.4.3.1 NGFC Support System (Sustainment Set-up) Personnel Operating Personnel
1.1.1.4.4.3.2 NGFC Support System (Sustainment Set-up) Personnel Maintenance Personnel
1.1.1.4.4.3.3 NGFC Support System (Sustainment Set-up) Personnel Supply Personnel
1.1.1.4.4.3.4 NGFC Support System (Sustainment Set-up) Personnel Training Personnel
1.1.1.4.4.3.5 NGFC Support System (Sustainment Set-up) Personnel Other Personnel
1.1.1.4.4.4 NGFC Support System (Sustainment Set-up) Data N/A
1.1.1.4.4.4.1 NGFC Support System (Sustainment Set-up) Data Technical Publications
1.1.1.4.4.4.2 NGFC Support System (Sustainment Set-up) Data Engineering Data
1.1.1.4.4.4.3 NGFC Support System (Sustainment Set-up) Data Management Data
1.1.1.4.4.4.4 NGFC Support System (Sustainment Set-up) Data Support Data
1.1.1.4.4.4.5 NGFC Support System (Sustainment Set-up) Data Data Repository
1.1.1.4.4.4.6 NGFC Support System (Sustainment Set-up) Data Other Data
1.1.1.4.4.5 NGFC Support System (Sustainment Set-up) Training N/A
1.1.1.4.4.5.1 NGFC Support System (Sustainment Set-up) Training Training Development
1.1.1.4.4.5.2 NGFC Support System (Sustainment Set-up) Training Initial Training
1.1.1.4.4.5.3 NGFC Support System (Sustainment Set-up) Training Training Equipment
1.1.1.4.4.5.4 NGFC Support System (Sustainment Set-up) Training Training Materials
1.1.1.4.4.5.5 NGFC Support System (Sustainment Set-up) Training Training Facilities
1.1.1.4.4.5.6 NGFC Support System (Sustainment Set-up) Training AMSE
1.1.1.4.4.5.7 NGFC Support System (Sustainment Set-up) Training AC Familiarisation
1.1.1.4.4.5.8 NGFC Support System (Sustainment Set-up) Training Other Training
1.1.1.4.4.6 NGFC Support System (Sustainment Set-up) Support & Test Equipment N/A
1.1.1.4.4.6.1 NGFC Support System (Sustainment Set-up) Support & Test Equipment Mission Support Equipment
1.1.1.4.4.6.2 NGFC Support System (Sustainment Set-up) Support & Test Equipment Test, Measurement & Diagnostic Equipment
1.1.1.4.4.6.3 NGFC Support System (Sustainment Set-up) Support & Test Equipment Other Support Equipment
1.1.1.4.4.7 NGFC Support System (Sustainment Set-up) Facilities & Infrastructure N/A
1.1.1.4.4.7.1 NGFC Support System (Sustainment Set-up) Facilities & Infrastructure Base Operating Facilities
1.1.1.4.4.7.2 NGFC Support System (Sustainment Set-up) Facilities & Infrastructure Maintenance Facilities
1.1.1.4.4.7.3 NGFC Support System (Sustainment Set-up) Facilities & Infrastructure Special Access Program Facility (SAPF)
1.1.1.4.4.7.4 NGFC Support System (Sustainment Set-up) Facilities & Infrastructure Other Facilities
1.1.1.4.4.8 NGFC Support System (Sustainment Set-up) Computing Support N/A
1.1.1.4.4.8.1 NGFC Support System (Sustainment Set-up) Computing Support MASIS / ALIS Interface
1.1.1.4.4.8.2 NGFC Support System (Sustainment Set-up) Computing Support Other Computing Support
1.1.1.4.4.9 NGFC Support System (Sustainment Set-up) Other Support System N/A
1.1.1.4.5 Operational Transition N/A N/A
1.1.1.4.5.1 Operational Transition System Assembly Integration, Assembly, Test & Checkout N/A
1.1.1.4.5.2 Operational Transition Contractor Technical Support N/A
1.1.1.4.5.3 Operational Transition Operational Test & Evaluation N/A
1.1.1.4.5.4 Operational Transition Conversion N/A
1.1.1.4.5.5 Operational Transition Sustainment N/A
1.1.1.4.5.6 Operational Transition Other Operational Transition N/A
1.1.1.4.6 Other N/A N/A

Cost Breakdown Sub-Structure – In Service Phase (1.1.1.5)
CBS# Level 5 Level 6 Level 7 (provided for understanding scope for this group)
1.1.1.5.1 Operation N/A N/A
1.1.1.5.1.1 Operation Personnel N/A
1.1.1.5.1.1.1 Operation Personnel Direct Personnel
1.1.1.5.1.1.2 Operation Personnel Support Personnel
1.1.1.5.1.2 Operation Operating N/A
1.1.1.5.1.2.1 Operation Operating Aviation Fuel
1.1.1.5.1.2.2 Operation Operating Unit Level Operating (Mission Support)
1.1.1.5.1.2.3 Operation Operating Other Operating
1.1.1.5.2 Support N/A N/A
1.1.1.5.2.1 Support Management N/A
1.1.1.5.2.1.1 Support Management Management Office
1.1.1.5.2.1.2 Support Management Agreements - PSFD MoU
1.1.1.5.2.1.3 Support Management Other
1.1.1.5.2.2 Support Unit Maintenance Support N/A
1.1.1.5.2.2.1 Support Unit Maintenance Support Consumable Matl / Repair Parts
1.1.1.5.2.2.2 Support Unit Maintenance Support Spares / Depot Level Repairable
1.1.1.5.2.2.3 Support Unit Maintenance Support Other
1.1.1.5.2.2.4 Support Unit Maintenance Support POL / Energy
1.1.1.5.2.2.5 Support Unit Maintenance Support Training Expendables
1.1.1.5.2.3 Support Depot Maintenance Support N/A
1.1.1.5.2.3.1 Support Maintenance Support Engine Maintenance
1.1.1.5.2.3.2 Support Maintenance Support Other Depot Maintenance
1.1.1.5.2.3.3 Support Maintenance Support Overhaul / Rework
1.1.1.5.2.3.4 Support Maintenance Support POL / Energy
1.1.1.5.2.3.5 Support Maintenance Support Training Expendables
1.1.1.5.2.4 Support Contractor Support N/A
1.1.1.5.2.4.1 Support Contractor Support Management
1.1.1.5.2.4.2 Support Contractor Support Engineering Support Services
1.1.1.5.2.4.3 Support Contractor Support ALGS Operation
1.1.1.5.2.4.4 Support Contractor Support ITC Operation
1.1.1.5.2.5 Support Programming Lab Support N/A
1.1.1.5.2.5.1 Support Programming Lab Support Programming Lab Operation (SW Maintenance)
1.1.1.5.2.5.2 Support Programming Lab Support Programming Lab Sustainment (Incl Updates)
1.1.1.5.2.6 Support Weapon Support N/A
1.1.1.5.2.6.1 Support Weapon Support Weapon Maintenance & Assurance
1.1.1.5.2.6.2 Support Weapon Support Weapon Replenishment
1.1.1.5.2.7 Support Training N/A
1.1.1.5.2.7.1 Support Training Simulator Operation (incl POL / Energy)
1.1.1.5.2.7.2 Support Training Simulator Maintenance
1.1.1.5.2.7.3 Support Training Continuous Training
1.1.1.5.2.7.4 Support Training Other Training Support
1.1.1.5.2.8 Support Support Equipment N/A
1.1.1.5.2.8.1 Support Support Equipment Support Equipment Maintenance
1.1.1.5.2.8.2 Support Support Equipment Support Equipment Replacement
1.1.1.5.2.9 Support Data N/A
1.1.1.5.2.10 Support Facilities N/A
1.1.1.5.2.11 Support PHS&T N/A
1.1.1.5.2.12 Support Sustaining Engineering Support N/A
1.1.1.5.2.13 Support Modifications (incl Procurement, Installation, Support, Obsolescence, update Data & Training, etc.)
1.1.1.5.2.14 Support Other Sustaining Support N/A
1.1.1.5.2.14.1 Support Other Sustaining Support Other Recurring Investment (incl Licences, etc)
1.1.1.5.2.14.2 Support Other Sustaining Support Other
1.1.1.5.3 Other N/A N/A
1.1.1.5.3.1 Other Upgrade (Including: Management, Development, Procurement, Installation, Support, Obsolescence, update Data & Training, etc)

Page details

Date modified: