Strategic Cost Estimation

The late New York Yankee player/philosopher Yogi Berra famously said, “It’s tough to make predictions, especially about the future.” This is very much the case in the security affairs of nations, as predicting the future actions and interactions of competing states is notoriously difficult. Regardless, national security policymakers are faced with the need to make decisions today that will have consequences for decades to come. To help inform those decisions, a particularly promising, yet under-utilized, approach is to focus on budgets as a means to understand potential strategic interaction among national security establishments. This article focuses on the use of analytical methods to forecast and bound the range of capabilities that foreign adversaries can feasibly procure over the medium- to long-term within their economic and technological constraints.

Because defense budgets by their nature possess some measure of opacity varying by nation, it’s a long way from “knowing what was spent” to “knowing what it was spent on.” Bridging this gap may be done by a specific application of cost estimation, which we henceforth call “strategic.” Applications of cost estimation that support strategic-level decision-making are what we call “strategic cost estimation and analysis.”

In an era of renewed great power competition, strategic cost analysis will be critical in understanding the resource constraints of the major powers and their ability to apply their resources in attempts to implement various national strategies. This article, the product of an ongoing project within the Center for Strategic and Budgetary Assessments (CSBA), explores methodologies for estimating defense-related cost constraints in order to better understand the potential bounds of the ongoing modernization efforts of the People’s Liberation Army (PLA) of the People’s Republic of China. More broadly, beyond a purely methodological discussion, this piece contributes to a dialogue in the profession about how mathematical models impact policy.

Defining Strategic Cost Estimation

The potential utility of cost estimation as a means of developing strategy is, in our opinion, currently undervalued. Largely in response to U.S. Department of Defense (DoD) requirements, the cost analysis field today is principally focused on programmatic cost analysis. Cost estimation is an integral part of the DoD acquisition process, with initial lifecycle cost estimates required to receive early (Milestone A) approval, accompanied by formal “Analyses of Alternatives” [1] based on capability needs [2]. There are many excellent resources [3] on traditional cost estimation.

But analyzing the cost implications and tradeoffs within a broader portfolio of programs at the enterprise level – what we term “strategic cost estimation” – has largely been overlooked in recent decades despite the feasibility and usefulness of such analysis for national-level decision-makers. The overall goal of strategic cost estimation is to determine how an external entity allocates resources based on relative costs in furtherance of an overarching national security strategy.

Similar cost estimation efforts have been pursued in the past, most notably U.S. government efforts to estimate Soviet defense costs during the Cold War. During that time, due to the opacity of Soviet economic data, strategic cost estimation grew as a discipline, and defense economics developed into a broad field. But this analytic method has been slighted in recent years, despite its promise for understanding the future capabilities of adversaries and the resulting value for U.S. defense strategists and planners. Understanding an adversary’s potential force structures – similar to determining a production possibilities curve in economics – is fundamental to evaluating and forecasting the medium- to long-term strategic balance between two or more states or actors.

As described below, a strategic cost estimate has several structural and methodological features. First and foremost, for truly opaque budgets, there is no “certificate” for solutions, in the sense that practitioners will never be presented with a “true” answer against which to grade their efforts. As illustrated in our case study on Chinese defense budget data that follows, much of the cost data required for strategic cost estimates of one’s competitors is unknown, likely even within the foreign government itself, due to the diversity of defense-related funding streams, inconsistent accounting standards and the inherent obscurity of cost data for complex weapons programs. Moreover, in many cases, cost data for adversary systems is intentionally withheld or obfuscated. Yet these difficulties are not insurmountable; there are meaningful, defensible approaches that practitioners can take, as set out further below.

Estimating U.S. Strategic Tradeoffs

Before trying to understand how other nations allocate their defense budgets, we first look to the United States. CSBA’s most recent contribution to strategic cost analysis is the strategic choices tool (SCT), originally launched in 2012. CSBA developed the SCT to focus on the tradeoffs confronting U.S. policymakers as they wrestle with constraints on U.S. defense spending. The tool allows participants to quickly and effectively explore the trade space of alternative investment and strategic choices facing the DoD over the next 10 years. Drawing from almost 1,400 options, teams invest in new capabilities and divest capabilities less critical to meeting their strategic aims given a specified level of resources. The options include major units of force structure, end strength, bases, readiness, civilian personnel, legacy weapon systems, modernization programs and technology development.

The holistic understanding of U.S. budgetary decisions and their impact requires the aggregation and smoothing of disparate data sources. Thankfully, DoD provides a wealth of unclassified, publicly available cost data on its recent and planned procurement of platforms and systems. In this case, the difficulty is not intentional obfuscation so much as inconsistent data standards. These data standards exist because of both internal and legal requirements and were not intended to be used in the way that we use them.

There is no single “right” answer to how these datasets should be merged and used, and the “right” choice depends heavily on context. In our use case, we purposefully choose our data with an eye toward holding wargames and exercises in which participants can use the SCT as part of budget-conscious evaluations of new operating concepts and alternative future force structures to support national security strategies.

Methodological Considerations

The key difficulty in cost estimates of foreign military platforms, systems and units is the lack of reliable and unadulterated data. Absence of data is the bane of any cost analyst but, paradoxically, these difficulties tend to give practitioners more freedom in making assumptions. When given a free hand in making choices based on equally compelling assumptions, we recommend the following principles, in which “explainability” is more important than low variance, which itself is more important than a precise point estimate.

1. Give explainability a greater weight than absolute accuracy. The topic of “explainability” is a large issue currently in the broader practice of operations research and statistics [4]. Understanding why a cost is estimated a certain way – and knowing what the true cost drivers are – is as or more useful to the decision-maker than having a precise point estimate. Therefore, an answer with high variability expressed as a wide confidence interval that can be explained is more valuable than a precise point estimate that comes from a so-called “black box.” Simple pictures are best.
   
   
2. Prefer minimizing variance to producing a precise point estimate. While the preceding point noted that “explainability” is more important than low variance, low variance is itself more valuable than a tight estimate. This is in keeping with the general principle of selecting a “minimum variance unbiased estimator.” When forced to choose, we prefer low variance over lack of bias.
   
   
3. Be aware – but not fearful – of so-called “mirror imaging,” the assumption that someone else’s processes are nearly identical to your own. This is perhaps the most delicate issue, as the systems we know the best are obviously our own. Other nations and/or political systems have different objectives, as well as different constraints. China’s central economy is able to make decisions about resource allocation that would not be possible in the United States and most of Europe [5], therefore broad assumptions about cost, infrastructure and intent are likely to be wrong. Nevertheless, given the lack of data, a degree of mirror imaging may be a useful first step, as long as the known data is still modified based upon the foreign military’s known operating procedures or system characteristics, such as operating hours or platform weight. It is here that a multidisciplinary team that involves domain experts outside of applied analysis is key, and perhaps it is most important that we are deliberate when mirroring.
   
   
4. In all assumptions where a “free hand” is given, take it – and try to be as broad as possible. Choose to focus on forward compatibility, asking yourself how would the model created by this method react if the data – or fundamental research question – were to change. Information about foreign military enterprises arrives either slowly, or all at once.

 The key to strategic cost estimation is the assumption that resources are finite and at some level, a defense program is a “zero sum game.” Policymakers, regardless of country, must make relative resource tradeoffs between different defense programs based on their overall basket of resources.

Strategic Cost Estimation in Context

Despite the significance of China’s rise in power and influence in the Asia-Pacific region and the world, surprisingly little is known about China’s defense economics and actual defense spending. Estimating defense costs for the People’s Republic of China is different – but at least as difficult – as doing so for the Soviet Union during the Cold War. Searching for analytical articles for “China Defense Platform Costs” on a web browser is useful only if your ultimate goal is to form a support group for frustrated analysts.

Due to the paucity of Chinese budget data, most analysts of the PLA have focused on analyzing the overall Chinese defense budget [6] rather than the costs of individual platforms, with only a few exceptions [7]. Much of the existing literature on China’s defense industrial base is descriptive; it highlights historical trends, prospects for, and technical capabilities of China’s defense industrial base [8]. Some analysts have focused on the capabilities of the defense industrial base in specific sectors, such as shipbuilding [9]. To date, there is no comprehensive assessment – at least in the public domain – of China’s industrial base that would allow analysts and policymakers to consider the opportunity costs of specific Chinese modernization investments and develop competitive strategies.

There is growing recognition within the policy community that the United States is in an era of great power competition with China [10]. Policymakers at all levels are hungry for information and insights that can help understand both China’s intentions and capabilities. Better understanding the hooks and levers of China’s national power, defense industrial base and resource tradeoffs may in fact stave off conflict by providing strategists with a more accurate understanding of the balance of power. This is our sincerest hope.

In the United States, there has been an unhealthy and unproductive tendency to focus on China’s strengths and implicitly assume that China’s allocation problems are unbounded. This is not likely, particularly as China’s economy slows and faces stronger headwinds in the coming years that may force hard budgetary choices within the PLA [11].

The importance of this problem is mirrored in its difficulty. China has been particularly opaque about its defense budget, although it has displayed varying degrees of transparency over the years. The degree of secrecy means that we are treating China as a true “black box.” China’s accounting methods are unlikely to be similar to those of the United States, which causes us to challenge our fundamental assumptions about nearly everything. As with the Soviet Union during the Cold War, it is possible – likely even – that China’s decision-makers themselves do not fully understand the true costs of their military.

The various subdisciplines that would advance our understanding of China’s defense decisions are stove-piped; PLA experts specialize in doctrine, organizations, institutions and strategy. There are vanishingly few PLA experts that explicitly specialize in costing. The necessary prerequisites for holistically studying China’s defense policy strategically are too arduous for a single person to hold; one would need to be a statistician who spoke Chinese with a degree in foreign policy. This is why interdisciplinary teams with broad perspectives that challenge the fundamental methodological assumptions are required.

Case Study: China Strategic Choices Tool

CSBA has already been making progress toward developing a China SCT. In developing cost estimates for Chinese defense systems, we evaluated both “bottom up” approaches – which estimate the levels of effort for each subcomponent of an industrial process, such as mining, milling, riveting, etc. – and “top down” approaches, where we plug the attributes of a completed platform or system into a statistically generated cost estimating relationship in order to produce a cost estimate.  

We make a broad assumption that: all large nations’ defense programs are multiyear, institutional behemoths that cannot pivot in a single year and are highly dependent on other programs; and that a nation’s spending on defense is a quantifiable expression of its strategy, aspirations and fears. Our tool functions in two variable time increments, usually set at five years each, which smooths out potential aberrations in certain years and reduces error introduced by varying accounting methods.

Importantly, in contrast with the U.S. government’s Cold War-era Soviet cost estimates, we are more humbly striving to develop accurate relative rough order of magnitude cost relationships between defense goods, rather than develop correct absolute costs for these goods. The SCT is not a budget-building tool; rather it is a tool for evaluating relative cost tradeoffs between platforms and units, a distinction that reduces the analytical burden of this undertaking. Relatedly, these estimates will not be used to generate an overall defense budget for China, nor will such a total budget estimate be used to estimate the percentage of China’s GDP spent on defense. These are currently overly ambitious goals for the first iteration of this project.

Our approach is as follows: First, understand what information is available about Chinese platforms; it does no good to build a statistical model based on regressors that do not exist. Broadly speaking, this is expressed in terms of basic platform characteristics, such as weight and power for ships and aircraft. Second, use these regressors to selectively build a cost estimating relationship (CER) for comparable Western systems (including learning [12]). We then take these CERs and apply them to the known (or estimated) characteristics for Chinese systems. Following these steps, we create the estimates shown in Figure 2.

This sample CER creates an estimate of Chinese aircraft costs in U.S. dollars, were they to be produced in the United States at U.S. industry production efficiencies. In the next stage of our work, we will apply a China-specific deflator, likely based on labor prices and productivity (determined from multiple sources, such as GDP per worker), to adjust these Western prices for the Chinese context. The cost figures that are ultimately produced will, of course, be inaccurate, but they will be useful enough for rough order of magnitude trades between different Chinese platforms. Moreover, similar to the U.S. government’s multidecade work on Soviet cost estimates, this is the first iteration of what will be a long-term effort and, as we receive feedback and, likely, blunt criticism, the estimates within this tool can be refined over time to better reflect the peculiarities of China’s defense industrial base.

Improving Understanding of Oneself and One’s Competitors

Understanding the resource constraints of both the United States and China will be crucial to evaluating the future power balances in the Indo-Pacific and determining how the United States can best develop defense concepts and capabilities that preserve peace and stability. To that end, the U.S. Strategic Choices Tool and the China Strategic Choices Tool can be used in interactive competitive exercises, featuring moves and countermoves between teams playing the roles of the United States and China, respectively, which will offer policy-relevant insights into the competitive dynamics between the U.S. and China.

Our project is at its midpoint. We are hosting a number of workshops and exercises over the next year in which we will evaluate and test our estimates and the new China SCT. If you are interested in attending, contact Jack Bianchi ([email protected]). For this complex, multidisciplinary project to succeed, we value informational and methodological input from a range of participants, from PLA experts and cost analysts to private sector competitive intelligence analysts and financial experts. 

Acknowledgment

The authors thank their colleague Toshi Yoshihara for his suggestions and insights early in their process.

References & Notes

1. https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/500002_dodi_2015.pdf
2. http://acqnotes.com/wp-content/uploads/2018/11/CJCSI-5123.01H-Charter-of-the-Joint-Requirements-Oversight-Council-JROC-and-Implementation-of-the-JCIDS-31-Aug-2018.pdf
3. https://www.cape.osd.mil/files/OS_Guide_v9_March_2014.pdf
   https://www.navsea.navy.mil/Portals/103/Documents/05C/2005_NAVSEA_CEH_Final.pdf
4. See, for example, Cynthia Rudin and Berk Ustun, 2018, “Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice,” Interfaces, Vol. 48, No. 5, pp. 449-466 (https://doi.org/10.1287/inte.2018.0957).
5. See Jingkuan Han, Yingjun Xu, Dingzhi Liu, Yanfang Zhao, Zhongde Zhao, Shuhui Zhou, Tianhu Deng, Mengying Xue, Junchi Ye and Zuo-Jun Max Shen, 2019, “Operations Research Enables Better Planning of Natural Gas Pipelines,” INFORMS Journal on Applied Analytics, Vol. 49, No. 1, pp. 23-39 (https://doi.org/10.1287/inte.2018.0974).
6. See, for example, “China Defence Budget,” Jane’s; Sean Chen and John Feffer, 2009, “China’s Military Spending: Soft Rise or Hard Threat?”, Asian Perspective, Vol. 33, No. 4, pp. 47-67; Shaoguang Wang, 1996, “Estimating China’s Defence Expenditure: Some Evidence from Chinese Sources,” The China Quarterly, No. 147, pp. 889-911 (September); Keith Crane, et al., 2005, “Modernizing China’s Military: Opportunities and Constraints,” RAND, pp. 91-134; Dennis Blasko, et al., 2006, “Defense-Related Spending in China: A Preliminary Analysis and Comparison with American Equivalents,” The United States-China Policy Foundation; Adam P. Liff and Andrew S. Erickson, 2013, “Demystifying China’s Defence Spending: Less Mysterious in the Aggregate,” The China Quarterly, pp. 805-830.
7. Richard A. Bitzinger, 2003, “The Challenge of Analyzing and Assessing Chinese Military Expenditures,” The China Quarterly, pp. 164-175; Gabe Collins, 2015, “How Much Do China’s Warships Actually Cost?”, The Diplomat, June 18.
8. For one leading work, see Tai Ming Cheung, 2009, “Fortifying China: The Struggle to Build a Modern Defense Economy,” Ithaca, NY: Cornell University Press.
9. For the most comprehensive assessment of one sector of China’s defense industrial base, see, Andrew Erickson, ed., 2011, “Chinese Naval Shipbuilding: An Ambitious and Uncertain Course,” Annapolis, MD: Naval Institute Press.
10. https://dod.defense.gov/Portals/1/Documents/pubs/2018-National-Defense-Strategy-Summary.pdf
11. https://www.foreignaffairs.com/articles/china/2019-03-11/whats-causing-chinas-economic-slowdown
12. “Learning” is the notion that over the history of production of most things, the effort required to build the subsequent one decreases over time, because you get “better” at it.