Next Generation Fighter Capability: Independent Review of Life Cycle Cost
3 Independent Review Observations
The independent review of the Estimate included an assessment of the scope, assumptions and calculations underlying the Estimate. The review criteria have been derived from relevant Treasury Board policies and other Government of Canada related instruments, applicable leading practices indentified in the Framework, and in consideration of the early Options Analysis phase of the NGFC project.
The key aspects that were reviewed are summarized below:
|Life Cycle Costing Framework Component||Review Criteria||Findings|
|NGFC Life Cycle Cost Planning||
|Model Data and Baseline Estimate||
3.1 NGFC Life Cycle Costing Planning
To support the development of the Model, at this phase of the project, we expected the life cycle cost project plan to outline key project elements, such as scope, purpose, schedule, data, costing methods and quality assurance. DND was not able to demonstrate that there was formal planning for the development of the Model. Although key elements of the plan were identified in other documents and within the Model, certain planning elements, such as schedule, purpose and quality assurance were not documented until well after the planning phase was completed. Planning is important to help ensure that appropriate resources, schedule dates and data collection activities are identified, authorized and allocated. In addition, a life cycle costing plan can help ensure the Model and Estimate is aligned to the purpose, with appropriate clarity with respect to the process to meet milestone, meeting and deliverable requirements.
It is recommended that DND formalize and document the life cycle costing plan in accordance with Framework guidance.
3.2 Model Boundaries
3.2.1 Ground Rules and Assumptions
The Framework identifies that all the key ground rules, assumptions and supporting documentation including technical, programmatic and acquisition strategies are fully documented and approved.
DND's documented key assumptions, Draft Ground Rules and Assumptions See footnote 4 include infrastructure requirements, structural life of the aircraft, training requirements, the types and number of aircraft to be procured and mission requirements.
Our analysis of the ground rules and assumptions found that the key assumptions were defined within the Draft Ground Rules and Assumptions, however certain key assumptions relating to yearly flying rate and fleet size required further interpretation to support costing as a result of rounding and wording ambiguity. With the exception of potential improvements to clarify these assumptions, project documentation definition is adequate at this stage to enable the development of a rough order of magnitude cost estimate. As the project moves forward through the Definition Stage and findings from proposed studies are analyzed, there is potential for changes in these key assumptions and further improvements to their clarity.
It is recommended that DND clarify documented assumptions with respect to yearly flying rate and fleet size and review and update the key assumptions and the Estimate on a regular basis and that agreed changes are reflected in the Estimate in a timely manner.
3.2.2 Cost Boundary
The Framework identifies that the boundaries of a Life Cycle Costing Model should be established to reflect three key aspects: the different purposes of the Model, the different information requirements related to those purposes, and the different cost elements relevant to those purposes. For the DND Annual Update to Parliament, the Framework indicates that the life cycle should include all costs out to and including the Program level. In addition, the span of years for the project and program reporting should include all incurred and estimated costs from the project initiation out to the disposal of all the NGFC.
We examined the purpose of the Model and the Estimate as provided to us by DND and compared these to the expectations established in the Framework. We found that the Model included costs out to the Program level, which thus included the contract level, the project level and the incremental level cost. The span of years for the reporting began at project initiation in 2010 and continued through to disposal, with an assumed lifespan of 30 years for each aircraft following its delivery.
Our review of the cost boundary did not identify any significant quantifiable differences in DND's application of the Framework. We have no recommendations related to cost boundary.
3.3 Model Structure
3.3.1 Cost Breakdown Structure
According to the Framework, "the Cost Breakdown Structure provides a logical and complete breakdown of the NGFC Program" See footnote 5. Our review of the Cost Breakdown Structure identified that the construct of the structure matched that outlined in the Framework and appeared to include all the main categories of cost elements relevant to the NGFC program. As all of the Sustainment costs are sourced from the JSF Program Office, the Sustainment costs component of the Cost Breakdown Structure aligns with the JSF Program Office structure. The Statement of Operational Requirement supplied to us was redacted, and thus we were not able to verify that all requirements were included in the Cost Breakdown Structure. For those areas we were able to review, we found that the Cost Breakdown Structure was generally complete. Cost elements that we could not assess for completeness included: diminishing manufacturing supplies, cost of certification of CF-18 weapons for the F-35s, security requirements upgrades to the deployed operating bases infrastructure, and changes in force structure, doctrine and tactics. We received a DND letter further summarizing DND's requirements and further discussed under "Other Potential Acquisition Cost", Section 184.108.40.206.
It is recommended that DND continue to review and update the Cost Breakdown Structure and the Ground Rules and Assumptions document to help ensure that the Cost Breakdown Structure and Estimate include all capability requirements.
The Framework identifies characteristics of a well constructed life cycle costing model. The Model was assessed against the Framework criteria including: accuracy, comprehensiveness, replicability and auditability, traceability, flexibility, credibility, and timeliness. Our model review focused on high risk areas, such as, inconsistent formulas in a series of formula cells, the application of sum total formulas, and keying errors relating to input variables provided by the JSF Program Office and by DND's Economic Model.
We identified the following findings:
- Multiple data entry – The Model requires that the same data be entered multiple times
- No logical flow of information – Financial models should follow a standardized flow such as:
- Control & Workflow Tab
- Validation Tab
- Input Tabs
- Calculation Tabs
- Output Tabs
- Input cell colour coding – Input cells in the Model were not always coloured differently from calculation cells or locked to assist the modeler in identifying calculation cells to prevent accidental changes
- The Model only forecasted 29 years (rather than 30 years) of Operating and Sustaining costs for the four fighter jets acquired in 2021. We understand this is based on the assumption that these four remaining F35s would not be repaired during the final year of service, but would rather be retired if service was required. Therefore Sustainment costs would not be incurred during this final year.
- DND used inflation rates from the 2011 Economic Model for Acquisition cost instead of the latest 2012 Economic Model inflation rates, resulting in a possible variance of up to $18 million in total Acquisition costs
- DND's forecasted costs was based on 11,700 flying hours per year, whereas fuel usage was incorrectly based on 12,000 flying hours per year. This results in a potential overstatement of operating costs by approximately $116 million.
While noting the above issues, the Model construct appears to have caused only limited errors in calculation. Other potential impacts of its construct include: inability to make timely updates and/or to perform sensitivity analyses, lack of auditability, and the potential of incomplete or inaccurate information.
It is recommended that DND refine and simplify the Model so that it better meets the Framework principles of flexibility, traceability, and ease of sensitivity analysis.
3.3.3 Cost Methods
The Framework identifies that each cost element should be costed using the most appropriate cost estimating technique.
The development of the costs related to the Project Management Office was undertaken internally using historical actual data. The personnel costs, which amount to over half of the Project Management Office costs, are detailed down to the specific numbers and rank of the individuals expected to be involved. There is a budget for studies to be undertaken during the Definition Phase which is appropriately expected to undertaken prior to Project Approval for the Definition Phase.
We found that the methodology used for the Acquisition costs for the project is a combination of engineering cost method and extrapolation from the actual costs of the first F-35 aircraft that have been produced. The engineering cost method involves direct estimation of a particular cost element by examining its components. These are appropriate cost estimating techniques for the project to use at this stage. The use of independent costing methods for the majority of the Acquisition cost component is not relevant at this time as most cost data comes directly from the JSF Program Office.
The project has used the estimates developed by the JSF Program Office for Sustainment. The methodology used for the development of the Sustainment estimates for the NGFC is mainly parametric analysis using actual historical data from US fighter aircrafts and using cost estimating relationships to derive an estimate for the F-35s. This methodology is the appropriate cost estimating techniques for the project to use at this stage. We understand that the next version of the United States Select Acquisition Report from the JSF Program Office will include some actual F-35 data based on the current F-35s in operation. Thus, as the project progresses towards Expenditure Authority for the Implementation phase, the cost estimates will necessarily improve.
The estimation of the Operating costs was undertaken internally by DND using historical actual. The cost methodology for estimating the Operating costs is based on an analogous approach using actual data from existing Canadian CF-18 support units/ bases. This is the most appropriate approach to use at the Options Analysis phase, considering the data that is available and the studies that are yet to be undertaken during the Definition phase which will further inform the Operating costs.
Our review of cost methods did not identify any significant quantifiable differences in DND's application of the Framework. We have no recommendations to cost methods.
3.4 Model Data and Baseline Estimate
3.4.1 Data Collection and Normalization
Based on the Framework, we expect that DND has collected data for all elements of the Program, from appropriate data sources, and normalized/adjusted them to allow for its appropriate use within the Model. We expect key costs and drivers are identified for further review and analysis, including sensitivity and risk. Our review process included determining the source of cost data used to develop the Estimate, its relevancy and currency, and how it has been normalized.
220.127.116.11 Treatment of Indices
Indices are used to account for inflation and are used for converting now year dollars (Constant Year) to future year dollars (Budget Year). Indices are also used for the conversion of one currency type to another. For the NGFC this conversion is mainly from US dollars (USD) to Canadian dollars (CAD). Currently some 95% of the Acquisition cost, and 40% of the Sustainment and Operations costs, are assumed to be in USD . Variation in indices can have a significant positive or negative effect on the costs.
The following data from indices are used in the Estimate:
- Foreign Currency Exchange (FOREX) is based on the Consensus Economic Inc. report which is published monthly. The Model uses the July 2012 Average Annual Rate of $1.062 US to CAD
- Inflation rates are based on:
- The DND Economic Model which is published by DND every year and is used to escalate military and civilian personnel and domestic costs for non JSF Program Office sourced costs
- A specific inflation rate which is developed for the JSF and is made available through the Canadian partner arrangement.
The Model uses the July 2012 Average Annual FOREX Rate which has not been updated to the most current long term forecasted rate. The potential impact of these different rates has been identified to be insignificant. For inflation, the JSF Program Office supplied rate provided to partner countries is around 1.8%. We understand that the true inflation rate is higher as components of 'inflation' are included within the base estimate. Based on information provided by DND, these inflationary figures appear appropriate See footnote 6 for this purpose.
Operating costs use indices developed by the DND and published in their Economic Model every year. The list of indices used is comprehensive and operating cost elements in the LCC Model use different indices from the Economic Model based on the type of activity relevant to the cost element. Fuel is a major component of the operating costs and is inflated at a rate of 4% per year based on the DND Economic Model. This is consistent with other planning undertaken within DND. Except as identified in Section 3.3.2 with respect to the use of an inflation rate that is not the most current, the source and application of the indices is consistent with the Framework.
It is recommended that the Government of Canada investigate mechanisms to more proactively manage foreign exchange risk for the NGFC Program due to the potential significant impact of FOREX on the Estimate.
The cost data for Development, which includes Project Management costs such as personnel, studies, accommodation, travel and other administrative costs, is derived using historical data drawn from other DND projects as benchmarks. These costs have been normalized and updated to the current price basis. The majority of the other Development costs are directly derived from signed Memos of Understanding.
For the Acquisition component of the Estimate, the JSF Program Office has developed the detailed cost estimates on which 90% of the baseline estimate is derived from. These costs are based on some actual cost data from F-35 aircraft in production and forecast on a learning curve to derive a US "Then Year" estimate of aircraft lot unit prices and support system costs. The cost data is based on the latest Selected Acquisition Report (SAR 11, December 31, 2011). As the various recent government organizations audit reports point out, there are still risks and uncertainties related to the JSF Program. It should be noted that we have not validated to the source cost data used in the Estimate, but rather have assessed whether the source of the data is relevant and current for the purposes for which the Estimate is developed and presented.
Acquisition costs have been developed to account for payment and delivery schedules based on JSF Program Office information. For example, payments for aircraft are provided in the following categories:
- Long Lead (5%)
- Full Funding (30%)
- Payment (45%)
- Delivery (20%).
These payments have also been aligned between US and Canadian financial years.
The cost data for the Sustainment component of the Estimate is sourced from the JSF Program Office cost data and is based on parametric methodology. Future Selected Acquisition Reports from the JSF Program Office will include some validation of cost from data received from experience on the F-35s that are now in operation in the US. The JSF Program Office cost data used for Sustainment in the Estimate is taken from SAR 11, which is the latest information that has been delivered to JSF Partners.
The cost data for the Operating cost component of the Estimate is sourced from DND's historical actual costs for the CF-18s from DND's financial database. The aviation fuel estimate uses information for the F-35 based on JSF Program Office calculations. Fuel is calculated using JSF Program Office supplied fuel rate per hour of flying for F-35s, using the DND endorsed fleet yearly usage and the cost per litre of fuel at Cold Lake base. The Operating cost estimate assumes that all current CF-18 personnel and bases will be used for the new Canadian F-35 fleet. While we agree with the general cost methodology and approach taken, we believe that further normalization/adjustment, in addition to fuel, could be made considering the differences between the CF-18s and the F-35s so as to further refine the Operating Cost estimate. These may include adjustments to reflect differences between the number of fighter jets, number of pilots, annual flying hours and differences in intermediate maintenance requirements. Due to the nature of these adjustments and based on DND's preliminary analysis, each of these impacts would likely reduce the current Operating Cost point estimate. Although detailed studies still need to be undertaken during project Definition Phase, preliminary studies would have been appropriate to consider the differences between the two aircraft and to develop a more informed estimate for the operating costs.
It is recommended that DND normalize and adjust all CF-18 Operating Costs to further refine the estimation of F-35 Operating Costs.
3.4.2 Baseline Estimate
Based on the Framework, and in consideration of the project status, our expectation is that the Estimate is complete, uses the most up-to-date information (see Section 3.4.1) and appropriate cost methods (see Section3.3.3).
In addition to the findings previously identified in the above referenced sections, the following outlines our specific findings in regard to the Estimate presented by cost category (Development, Acquisition, Sustainment and Operating) and cost element.
18.104.22.168 Development Cost
The cost estimates within Development include:
- Systems Development and Demonstration
- Production, Sustainment and Follow-on Development
- Project Management cost to Expenditure approval.
Some of these costs have already been expended in support of Canada's involvement in the JSF program.
The remaining funds are primarily to support the Production, Sustainment and Follow-on Development along with some 25 studies to be undertaken including Concept of Operations, Concept of Supply and Maintenance, Training, and Environmental noise analysis. Due to the limited materiality of these costs and the available time, these costs were not further investigated and no significant quantifiable differences in DND's Development cost estimate were identified.
22.214.171.124 Acquisition Cost
The cost estimates within Acquisition include the funding required up to the end of the Implementation Phase of the program. The primary driver of acquisition cost (71% of $8,388 million) is the Unit Recurrent Flyaway (forex) cost that is composed of the following key elements:
- Vehicle Systems
- Mission Systems
- Propulsion Systems
- Engineering Change Orders.
At the time of writing, consistent with similar projects at the Options Analysis phase, no approved master schedule was available for this Program. We understand that during the next phase of the NGFC project a more detailed schedule will be developed following a series of studies designed to provide greater understandings of interdependencies and activities that will need to be completed. We therefore depended on schedule and key dates information from the Project Charter, draft Project Management Plan for Definition Phase See footnote 7 and the NGFC Project Management Plan See footnote 8. Schedule alignment is a component of Acquisition cost as it identifies when planned acquisition will take place and at what estimated cost.
Unit Recurrent Flyaway Costs
Canada, and other partners, are solely reliant on the JSF Program Office for unit costs. The latest estimates now incorporate knowledge from actual construction costs of some 25 aircraft See footnote 9 that have been delivered. Based on the currently projected order profile for Canada and JSF Program Office unit costs, the weighted average US price is approximately $87.4 million USD in Budget Year. This weighted average unit cost of the F-35 is reflective of a confidence level of approximately 50% See footnote 10, which is typical of a baseline estimate.
It should be noted that the confidence level may not reflect all acquisition risks specific to Canada. As an example, additional risks related to potential future changes in the production demand profile and foreign exchange could impact Canada. In other words, to achieve a 50 percent confidence level for the Canadian URF cost estimate, additional contingency to reflect Canadian risk factors may be required.
Our review of the URF cost estimate did not identify any significant quantifiable differences in DND's application of the Framework.
The Canadian F-35s being considered would be delivered with Block Upgrade 3 as part of the Acquisition costs. We understand that DND will participate in all future Block Upgrades. The costs for these future Block Upgrades have been included in the Sustainment component of the Estimate in line item "Overhaul/ Rework". This cost data has been supplied by the JSF Program Office and is the most current supplied to JSF Partners.
DND has undertaken site surveys of bases that will be affected by the introduction of the NGFC to refine their infrastructure costs. The estimates presented use a DND infrastructure standardized methodology and cost template. There are 23 renovation and construction sub-programs that make up the NGFC Infrastructure program. The cost estimates have been developed internally by DND staff that specialize in Defence infrastructure. The cost estimates include construction and/ or renovation costs for facilities, such as hangars and taxiways, design costs, project management, travel and other administration costs, site security and personnel staff costs.
The cost estimates appear comprehensive and have been developed using standard fees and rates for construction and design fee costs, as well as high level construction and renovation cost data from previous projects. At this stage these cost estimates are regarded in the DND documentation See footnote 11 as "rough order of magnitude". It is understood that the next stage in the development of the estimates for NGFC infrastructure is to acquire the services of an external quantity surveyor to undertake more detailed studies. Our review of the infrastructure cost estimate did not identify any significant quantifiable differences in DND's application of the Framework.
Other Potential Acquisition Cost
In addition to the cost identified above and our review and comparison of the Cost Breakdown Structure, KPMG requested and received a DND letter further summarizing DND's requirements, assumptions and cost treatment of the drag chute, air-to-air refuelling, weapons and NORAD See footnote 12. With respect to the drag chute and NORAD , KPMG received confirmation from DND that the F-35 meets the mandatory requirements documented within the Statement of Requirement without modification and without additional cost See footnote 13. With respect to weapons requirements, KPMG received confirmation that weapons currently in DND inventory, which can be employed on the F-35 fleet, will be retained and the initial stock of other weapons requirements related to gun ammunition and countermeasures had been included in the Estimate. We understand that the acquisition of newer weapons will be considered and funded as separate projects and comparison with Australia would suggest that these costs could be substantial (greater than $1 billion). With respect to air-to-air refuelling requirements, DND will rely on NORAD , coalition partners, or commercial refueling assets to meet operational requirements, and thus, based on these requirements and related assumptions it would not be appropriate to include potential asset modification costs in the Estimate.
In addition to the original acquisition cost, another potential acquisition cost relates to attrition aircraft. To assess whether replacement of attrited aircraft is a requirement, a key guiding document is the Government endorsed Canada First policy that states that:
"Starting in 2017, 65 Next Generation fighter aircraft to replace the existing fleet of CF-18s. These new fighter aircraft will help the military defend the sovereignty of Canadian airspace, remain a strong and reliable partner in the defence of North America through NORAD , and provide Canada with an effective and modern air capability for international operations". See footnote 14
If the F-35 is acquired, we understand that aircraft are anticipated to be lost (i.e. damaged beyond economic repair) due to accident or other events. This type of loss is commonly referred to as attrition. Within the military context, attrition is normally estimated based on the total flying hours of the Program. Based on a fleet of 65 F-35 aircraft with a steady state annual flying program of 11,700 hours, DND have estimated the potential range of attrition to be as low as 7 aircraft or as high as 11 aircraft over the 30 year life See footnote 15. We understand the Office of the Auditor General previously reported attrition could be as high as 14 aircraft See footnote 16, based on a higher flying hour program over 36 years. The final estimate will be further refined when the flying hours program is finalized, which is planned to be completed as part of the next stage of the development process.
If these attrition aircraft are not replaced, then over time the capacity of the DND to generate operational capability from the JSF would naturally decline. That is, if 65 aircraft are the minimum number to meet the requirements of Government, then over time DND would either no longer be able to meet these requirements, or would only be able to meet them with increased risk. This is presented in a stylized manner below:
Figure 1 - Aircraft Attrition Risk
Figure 1 - Aircraft Attrition Risk – Text versions
The current operating requirements, as identified in the Statement of Operational Requirements, identify the requirement to either purchase up front, or purchase at a later date, attrited aircraft. While attrition aircraft may not form part of the Project level cost, as outlined in the Framework, the Estimate should include all expected costs to maintain 65 aircraft capability at the Program level over the life of the NGFC capability. Additionally, not including attrition is an applied assumption that is inconsistent with the assumption of including the operating cost of 65 aircraft for 30 years, currently within the Estimate.
To quantify potential attrition cost to maintain a constant fleet size of 65 aircraft, we have undertaken a high level analysis using the minimum and maximum numbers discussed above. A minimum cost estimate is seven aircraft with a weighted average cost of $92.1 million ($87.4 million USD ), plus 13.5% contingency, giving a lower range of $0.7 billion. At the upper band, 11 aircraft using the same weighted average cost and 30% contingency would provide for a total estimate of $1.3 billion. DND's provided attrition estimate is $982 million and is within the KPMG identified range above.
DND have disclosed potential attrition in the Annual Update as a separate line item within the Estimate, in order to clearly identify the cost and recognize that the cost is outside the project scope due to current DND guidelines with respect to the treatment of attrition. Any attrition-related cost/decision will be made in the future, given that timing and replacement URF cost is uncertain and future capability requirements and replacement options may change. We understand that a replacement decision will depend on a future Government decision.
126.96.36.199 Sustainment and Operating Costs
The Sustainment costs cover the contractor supported on-going maintenance and repairs to the F-35 fleet and support systems, such as the aircraft simulators over its life until disposal. Sustainment is estimated at approximately $13.3 billion over 30 years life cycle. The key components of the Sustainment costs are:
- Unit Level Consumption covers consumables and repair parts as well as depot level repairs. The estimate over its life of 30 years is $5.3 billion (Budget Year).
- Depot and other covers major overhauls and maintenance at centralized facilities. The estimate over its life of 30 years is $0.8 billion (Budget Year).
- Contractor support covers technical and training support including maintenance of training equipment and centers as well as global supply chain management. The estimate over its life of 30 years is $2 billion (Budget Year).
- Sustaining and other support covers block upgrades to the F-35 fleet, software maintenance, the management and support of the Programming Lab which will be used to program specific Canadian software and simulator support. The estimate over its life of 30 years is $5.2 billion (Budget Year).
The above estimates are all developed by the JSF Program Office using mainly parametric analysis methodology and supplied to DND on an annual basis.
Approximately forty-two percent of the Sustainment costs are variable and linked to the planned Yearly Flying Rate which in the case of this project is 11,700 See footnote 17 flying hours per year for the fleet. Therefore the Yearly Flying Rate is the major cost driver for the Sustainment costs. DND's sensitivity analysis shows that increasing the Yearly Flying Hours to the current CF-18 rate of 15,800 hours would increase the Sustainment cost by an additional $1.8 billion (Budget Year).
Operating costs cover the costs for aviation fuel for the aircraft, personnel salaries and benefits, maintenance and repairs to bases and training costs and are estimated at approximately $20 billion (Budget Year) over the life of the aircraft fleet. The key components of the Operating costs are:
- Personnel costs which are estimated at approximately $10 billion (Budget Year) over the life of the aircraft fleet
- Operating costs which includes aviation fuel, unit level operating costs and support costs for the bases which is estimated at approximately $9.7 billion (Budget Year) over the life of the aircraft fleet.
Approximately twenty-five percent of the costs are related to fuel usage, which in turn is related to flying hours for the fleet. DND's sensitivity analysis shows that the impact on Operating costs of increasing flying hours to the CF-18 rate, which is 4,000 hours more per year, is an increase of approximately $1.5 billion dollar over the life of the fleet.
Operating costs have been developed by DND using the CF-18 data. As described in Section 3.4.1, further analysis to normalize/adjust for CF-18 and F-35 differences could be done.
Residual Life of Aircraft Total Flying Hours
The current Estimate is based on a flying program of approximately 11,700 hours per year. Based on a fixed flying program of 11,700 hours per year, 65 aircraft F-35 fleet would have an average age at retirement (30 years) of only 5,400 hours. Considering the current airframe structural life is approximately 8,000 flying hours, each plane on average would have approximately 2,600 hours, or one third, of its structural life remaining at the end of 30 years. It should be noted that this flying hour calculation does not include an allowance for operational hours.
The above highlights the potential residual life of the F-35 fleet on retirement which should be studied to help ensure its potential use is adequately reflected in future estimates.
It is recommended that DND conduct further analysis, and communicate key assumptions, in regards to the effective use of the remaining aircraft life at the end of 30 years.
188.8.131.52 Analysis of Disposal Cost Estimates
The cost estimates within Disposal include the funding required to dispose of the F-35 at the end of its life. It includes the removal of F-35 from service and retirement of any potential financial liabilities. Disposal cost includes management, studies and analysis and the actual disposal activities considering security arrangements, environmental, safety and occupational health aspects.
At this stage in the life cycle of the NGFC Project and for the purposes of the Annual Update, the Disposal Estimates are expected to be rough order of magnitude quality. The estimates used to develop disposal costs are based on data from the US Government Accountability Office (GAO) Report: "DoD's Liability for Aircraft Disposal Can Be Estimated", November 1997. The GAO gives disposal costs for the CF-18 unit cost, an average fighter unit cost and higher end fighter unit costs in 1997 US dollars.
The DND Estimate is based on the higher end fighter unit cost, increased by a further 33%, inflated to 2012 price basis and converted to Canadian dollars. The demilitarization, storage and removal of hazardous material are included in the cost estimates. First year of expenditure on disposal costs is planned for 2045 and the disposal costs are estimated to be approximately $43 million. There is a significant amount of uncertainty related to disposal costs, but this is not surprising considering the timeframes for which disposal costs are needed. DND has included additional contingency of $22 million to reflect these uncertainties.
The methodology used and data source are appropriate for the development of the disposal cost for the NGFC at this stage. Data has been appropriately normalized and inflated to current price basis. Further work over the lifetime of the NGFC will improve the cost estimate. Our review of the disposal cost estimate did not identify any significant quantifiable differences in DND's application of the Framework.
3.5 Estimate Results
3.5.1 Sensitivity Analysis
The Framework states that sensitivity analysis be undertaken and that the results be documented and communicated. Sensitivity analysis is a very useful tool for aiding in the management of uncertainty and project cost risks and informs decision makers as to the confidence they may have on the Estimate presented.
We found that DND had carried out a series of sensitivity analysis studies. A classified study, "The Unit Recurring Flyaway Cost of a Canadian Joint Strike Fighter" (November 2011), examined the potential impact of international withdrawal or downsizing of fleet numbers. Another classified study, "Forecasting National Procurement Costs for the Joint Strike Fighter" (May 2012), examined the potential cost of operating and maintaining the Canadian F-35s over their life cycle. Both of these studies were undertaken by DND's Centre for Operational Research and Analysis. Further studies have examined changes in indexation (inflation), foreign exchange rates and varying assumptions related to F-35 unit costs. These studies appear to cover an appropriate range of possibilities and likely key drivers.
3.5.2 Risk and Uncertainty Analysis
As derived from the Framework, and in consideration of the current project stage, we expect DND to have undertaken an analysis of risks and uncertainties. From this analysis we would expect DND to have included in the Estimate a budget to mitigate identified risk and uncertainty - commonly referred to as contingency. We would anticipate that this contingency would be commensurate with the assessed level of risk to achieve a desired level of confidence in the Estimate. As an example, if the desired confidence level was 65%, it would mean that there is a 65% chance the project would be delivered at or below the Estimate and a 35% chance it would be delivered above the Estimate. As outlined in the Framework, appropriate confidence levels range between 50% and 90% depending on user needs and the risk appetite of the organization. In effect the Estimate will be a risk adjusted estimate that includes the necessary contingency to bring the Estimate to an appropriate confidence level.
The DND NGFC Project Charter states that the overall risk assessment for the project is "High" See footnote 18. High or significant risk ratings have been identified for risk of incomplete planning assumptions, schedule, cost uncertainty, technology and scope. Given DND's "High" ratings for the core elements of the project – cost, capability and schedule – we would expect that the Estimate include a sufficient contingency budget to account for these risks. We examined all major cost elements, associated documentation and studies to determine whether the level of contingency was reasonably established.
In order to develop an understanding of what the final risk adjusted costs might be, we examined existing DND documentation and the Framework to develop a range of contingency estimates for Acquisition, Sustainment and Operations.
The primary cost driver of Acquisition cost is the unit cost of the F-35. Based on advice from JSF Program Office, the average unit costs of the F-35 to Canada is currently approximately $87.4 million USD (Budget Year). However JSF Program Office is only approximately 50% See footnote 19 confident that actual costs will be at or below this level. There still exist a number of risks that may result in further increases to costs and/or may additionally impact on Canada's acquisition cost, for example:
- the inability to generate the level of learning underpinning reductions in URF costs
- other changes in production costs
- changes in order quantities
- risks associated with managing critical concurrent activities
- software development risks
- further engineering changes to achieve required performance
- foreign exchange issues
- price indexation (inflation).
Based on DND developed modeling See footnote 20 of the unit price of the F-35, a 13.5% contingency would be required to derive a 55% level of confidence. As a result, this is identified to be the minimum level of the contingency range. Given a total base estimate of $8,388 See footnote 21 million, 13.5% would be approximately $1,132 million.
In addition to the above analysis, DNDs general assessment See footnote 22 of project contingency states that even where a project has substantive costs (effectively contractually valid costs), accuracy of estimates could be underestimated by as much as 15% to 20%. The current program has not yet reached a substantive level of cost certainty. Based on this analysis, and our judgment, the outer range of the contingency should be higher than 20%. In Australia, the default level of contingency for similar High risk cost elements in major Defence projects is 30%, with the final level of contingency derived based on an analysis of the specific risks and uncertainties associated with each cost element. Based on the current risk profile, DND practice and other comparative analysis, the maximum range of applicable acquisition contingency for this purpose is considered to be 30%. Given a total estimate of $8,388 See footnote 23 million, 30% would be approximately $2,516 million.
Based on an analysis of the current DND acquisition estimate, the contingency required to establish an appropriate risk adjusted estimate would range between $1.1 billion and $2.5 billion. DND's current contingency provision of $602 million neither falls within this range, nor does it meet DND's estimates for contingency of $1.5 billion See footnote 24 based on current assessment of risk.
DND has advised that their risk mitigation strategy for Acquisition costs, to remain within a $9 billion ceiling, is to reduce the number of aircraft acquired. As a result, based on their own calculations of potential contingency required, this could reduce the initial fleet to as low as 55 aircraft See footnote 25, which is below DND's current stated requirements.
Sustainment and Operating Contingency
The JSF Program Office has adjusted the Sustainment costs for growth above inflation See footnote 26. They also state that given the significant increase in capability, it is not unreasonable that the F-35 would cost more to operate and sustain than certain legacy aircraft See footnote 27. The DND Sustainment costs are therefore supplied by the JSF Program Office with a level of risk adjustment. Nevertheless, the estimates are based on parametric costing methodology so there is still a significant level of uncertainty and risk related to these JSF Program Office Sustainment estimates. Some countries do not allocate contingency against the sustainment and operating costs but they do make an allowance for risk in their baseline estimates. For this reason, the minimum range of potential sustainment contingency has been assumed to be 30 percent.
A previous DND NGFC Cost Estimate stated that given the parametric modeling approach used to develop Sustainment costs "Defence would typically consider it prudent to assign a contingency of 15% to a sustainment estimate of this quality at this stage". It is important to recognize that some of the efficiencies projected for the F-35 relying on the introduction of Automated Logistics Information System (hardware and software) have not yet been demonstrated See footnote 28. With Sustainment costs of approximately $13,290 million, 15% contingency represents approximately $1,993 million.
Based on currently available information and limited study undertaken by DND, it is not possible to quantify potential risk and uncertainty relating to Operating costs. It is feasible that savings might be generated due to potential reductions in manpower and related costs based on a reduced fighter fleet – moving from 77 CF-18s to 65 F-35. Also, savings may result from changes in how the F-35 is planned to operate, such as not requiring intermediate maintenance.
Based on the above, the current DND Sustainment estimate of contingency, to establish a risk adjusted estimate might be expected to fall between $zero and $2 billion and no range of required contingency for operating cost is provided.
DND have included a provision of approximately $1.95 billion within its estimates for Sustainment Contingency. Our review of the sustainment contingency cost estimate did not identify any significant quantifiable differences in DND's application of the Framework.
Summary of Contingency Analysis
Overall, DND has identified a total risk provision (contingency) of approximately $2.6 billion, and we have identified an expectation of a risk requirement (contingency) of between $1.1 billion and $4.5 billion. At a whole of NGFC Program level, therefore total contingency for the current level of risk and uncertainty within the F-35 Program is consistent with the Framework.
It is recommended that DND allocate an appropriate level of contingency to Acquisition cost, to reflect the remaining acquisition risks and desired level of cost certainty.
3.5.3 Document Results
The review of the cost Model and related findings is included in the preceding sections of Section 3. The product review findings of the Annual Update are included in Section 3.6 (Report Structure) below.
3.5.4 LCC Assurance
Our expectations as specified in the Framework are that the Model and the Estimate should be independently assured prior to any major milestone or in a manner consistent with the plan. This assurance activity could be in the form of an independent review and/or the development of an Independent Cost Estimate. The primary purpose is to challenge the existing Estimate to help ensure it is robust and reliable, taking into account the current life stage of the project and knowledge of the system under investigation.
We believe that this Report satisfies the requirement for independent review and that it is aligned with the practices in other JSF Partner nations, such as Australia, the Netherlands and Norway.
Based on the findings noted above, there are no recommendations related to the conduct of life cycle cost assurance for this Model and this Estimate.
3.6 Report Structure
The two reports reviewed and assessed in this section were the Model, assessed above, and the draft Next Generation Fighter Capability – Annual Update See footnote 29 (received by KPMG on November 20, 2012), assessed in this section.
Our expectations with respect to the report, in consideration of the Framework, were that the Report structure and results were appropriate for the purpose to support information for decision making, including the use of a standard life cycle cost analysis report structure to bring out key issues related to the Estimate in a concise, factual and easily understood manner.
The draft Annual Update provides a comprehensive overview of the key issues and risks associated with a potential F-35 program. The report outlines a range of analysis that has been undertaken by DND on current costs and risks and provides the basis and source of a number of these elements. DND presents a range of factors to provide greater clarity of the Estimate and related context. We provide the following additional points and feedback with respect to the Annual Update:
- We believe the acquisition costs are better than a "rough order of magnitude" and are of suitable quality to support the next stage of Government decision-making.
- The current disclosure on sensitivity analysis could be better integrated into the presentation of the Estimate.
- Date modified: