Wargaming Cybersecurity

The office looks like a computer games hobbyists’ convention: young men and women intently follow the action on computer screens, hunting for threats – advancing forces, sneaky infiltrators, hackers trying to disrupt the computers’ communications – and trying to counter them. But this is no game. It’s the reality of modern military conflict.

The current widespread interest in cybersecurity has led to lively discussions among experienced wargamers: How can we game cyber concerns? How much of this can we do at an unclassified level so that participation is not limited to a few experts, probably with strong points of view that can impede innovation? And can we do it quickly enough to be of practical use?

One big obstacle is adjudication. We know the outcomes of actual battles fairly soon afterward, and we know in general what armaments will be effective against which targets. The effects of a currency change or interference with a communications system may not be clear for years – or perhaps not at all. But if some of these challenges can be met, wargaming offers the same huge benefit in cyber applications that it does in kinetic conflict: the ability to test many “what-ifs” about moves and counter-moves, identifying vulnerabilities and preparing players to make good decision quickly in real conflict, all without the prohibitive cost of actual conflict.

A second obstacle is the highly classified nature of cyber systems, and hence gaming cyber systems. The level of classification means that we limit our use of the intellectual capital of the United States and allies and put at risk our ability to gain edge over our adversaries. We must find ways to wargame cyber at the unclassified level while dealing with information security dangers to best use the skills within academia, business and the gaming community [2].

Proposal: Use in-stride adjudication to game cyber effects at the operational level of war.

The most recent Connections 2018 USA wargaming conference, held in July at National Defense University, Washington, D.C., featured several working group discussions about these topics. Two experienced wargamers, Merle Robinson, a retired DoD analyst, and Stephen Downes-Martin, a research fellow with the U.S. Naval War College, ran a working group on real-time, or in-stride, adjudication. The working group identified recommended addressing the trade-off between speed, accuracy, flexibility and quality of adjudication to maintain the credibility of the adjudicators and the wargame results [1]. This implies the possible utility of using wargame techniques to game this design trade-off, using games with different designs in the speed, accuracy, flexibility and adjudication quality and testing for credibility of the adjudicators in the minds of the players and of the results in the minds of external assessors.

An analogous situation occurred in the early 2000s when early simulation experiments in operational-level information security concluded that with too little security, you end up dying, but with too much, you fail because you can’t move quickly enough. There’s a sweet spot somewhere in the middle, its location unknown. Wargaming is a good way to create hypotheses to feed those simulation experiments.

One of the proposals made during the Game Lab “How can we credibly wargame cyber at an unclassified level” at Connections USA 2018, was described as follows:

“Focus on the generalized characteristics and effects of cyber and information weapons independent of the detailed characteristics of specific cyber weapons. One can usefully research and explore the effects and wargame the operational usefulness of characteristics of a kinetic weapon without having to use the classified details of a specific example of that weapon. Similarly, it may be useful to wargame the usefulness of generalized cyber capabilities to explore the possible operational value and dangers of those capabilities and of the relationships and trade-offs between those characteristics.

“For cyber, however, the characteristics may look very different, and it is these differences that are interesting. For example, artillery rounds remain effective no matter how many have been fired in the past. Their trajectory, time on target and point of impact (CEP), weight and effects are important considerations and are determined by well understood physics. The equivalent characteristics of cyber weapons are very different. They often become ineffective after first use (the adversary can make themselves invulnerable to them), their trajectory through cyberspace (the Internet) is unknown (raising interesting political issues), time to become effective and the effect itself are highly uncertain, and collateral damage may be uncontrollable. Nevertheless, wargaming weapons with these general and unclassified characteristics is possible and valuable for tactical, operational, strategic and planners” [1, Section 3.7]; see also [2, 3].

Limitations of Gaming: Where Analysts Add the Most Value

One particularly strong warning about the limitations and appropriate application of wargaming comes from Ed McGrady, long-time head of the renowned wargaming group at the Center for Naval Analyses, and now an independent wargaming consultant working mostly with energy utilities. McGrady was part of the In-Stride working group.

“For technical cost-benefit questions,” McGrady asserts, “gaming is the worst way to find the tradeoff. You have to do analysis, looking at the characteristics of the system and what you’re trying to accomplish. For example, for alternatives in physical access, you propose sets of criteria for access and ask: How expensive is that compared to what you’re willing to spend on the system to keep it running? If you have to have a double for every person, to preserve every function, you’ve doubled your personnel costs. Is the added benefit worth it? But that’s just analytical. There’s no game involved. It’s more a risk analysis. Having an engineer assess vulnerabilities is better than trying to get at those vulnerabilities with a game.”

McGrady adds, “Whether a missile hits the target is an engineering question. If you introduce technical issues that undergird cyber into a game, players and, especially, adjudicators don’t have the technical knowledge to assess what will happen.

“What can you get out of a cyber game that you don’t get with a systems analysis? If you want the implications of what happens if a turbine fails because of a cyberattack, that can be gamed, but that’s cyber as an element in a game, not a game all about cyber. The cyberattack has to be assumed to succeed, and then you look at the effectiveness of proposed responses.”

Instead of trying to game attack effects, McGrady says, “I would talk to the client’s cyber experts and understand their system. Most of these organizations have been burned multiple times by people coming in and saying, ‘Sure, we can game that!’ without getting the system right. I would advise the sponsor to put the money on analysis rather than that sort of gaming.”

According to McGrady, it is worthwhile to game decisions and consequences. “You can do a tactical game on attacks on networks – have them apply a set of tools to the specifics of their system, then game attacks to help decide whether those tools are worth the cost,” he says. “Now the game becomes more like kinetic combat: How secure is my system against malware attacks? If I’m the attacker, how secure is my malware against rogue packets? As defender, how much do I want to pay for securing my system? The best way to secure a server is to seal it in concrete and drop it into the ocean, but you don’t get much value out of it then.”

And, he notes, “Shenanigans – low-probability, high-cost events – are difficult to deal with in a game. Instead of a game, I should be doing penetration testing.”

In government, he points out, risk is the biggest cost. “If you succeed, nobody notices, but if you fail, you probably get fired. In industry, though, companies succeed by accepting some level of risk, and they focus on expected gain.”

So, the objective of wargames and analyses is different. And that’s where the analysts come in. Engineering-based cost-benefit analysis appears to be essential to in-stride adjudication of games in domains like cyber conflict. Note, however, that contributing effectively requires both engineering expertise, at least somewhat specific to the domain, and some acquaintance with wargaming, especially the adjudication processes.

Recommendation: Use a Systems Approach Combining Qualitative and Quantitative Elements

Lt. Col. Michael Bond, a staff member of the Directorate of Strategic Plans and Programs (A5/8), Air Staff, U.S. Air Force, weighs in mostly in agreement with the view that wargaming should not be used as a substitute for analysis. “The Naval War College considers models, simulations and other tools to be labor-saving devices for adjudicators and sometimes players,” he says. “The same may apply to wargaming tools for cyber operations. One approach could be to use the systems thinking model to consider cyber actions and effects. Systems thinking is a precursor to a system dynamics model. The latter may be directly modeled” (see [4]).

Bond suggests creating a qualitative model first, then moving on to create a quantitative model – another skill set many OR/MS analysts would do well to learn, or to learn better.

Bond explains, “There are often a number of actions that must take place before a security incident occurs, each with a certain probability. If the incident still occurs, there are also a number of actions that must take place for the incident to have negative consequences. The diagram has a bowtie shape, as the risks narrow down and the consequences expand. This method can be used in cyber wargaming to model the attack surface, or all the potential risks that lead to a compromise, and the potential consequences. Once the various attack models are developed, the system could potentially be automated to some degree.”

In addition, he points out, “Regardless of the tool employed, it is important to note that cyber actions must match the time scale of the game. For example, if it takes several weeks to take a cyber action and a little more time to receive feedback, but the game time is only days, then it is senseless to try to play out that action.”

Hey, Not So Fast: Technical versus Operational/Policy Issue

The discussion at Connections 2018 was lively, as the reports only partially reflect, and the topic of gaming cyber had to address the difference between using games to address technical, operational and policy issues. For example, as Downes-Martin pointed out, “Many people are gaming cyber in the commercial world. Brown University, for one, has a course in security in cyber for executives. They don’t need the technical details. It’s all about gaming to get people to understand they have a problem. They look at crisis management, press relations, resiliency. They have gaming techniques that might be of interest in the government sphere.

“In addition, it’s interesting to examine what you won’t see about defending yourself if you don’t consider refining how well the adversary can employ cyber against us and how badly we might perform. For example, before World War II, the U.S. shut down analysis of propaganda because we find it distasteful [5, 6]. So, we missed much of how to respond. The British maintained a conniving tricks department. As a result, they made good inferences about what weapons the enemy was and was not developing, based on what was being said and not said by their propaganda ministry.

“Of course, this area is squishy and unknown,” Downes-Martin concedes, “but that’s why using both gaming and sensitivity analysis can be helpful. You do the game at an inductive level where you’re capturing the adjudicators’ and players’ reasons for their decisions.”

And review of such experiences can indicate which topics would be most beneficial to analyze in more detail.

Another under-explored area is the extent to which certain designs intended to enhance security might actually undermine it by increasing the chances for software design errors that would cause the same sorts of damage as cyberattacks. The most damaging “attack” on U.S. information and communications systems to date, at least among those known in the unclassified realm, was probably the crash of the U.S. long-distance telephone system in 1990, perpetrated by a minor error in the improved code AT&T had introduced to route calls. Experts warned that with the increasing complexity and software-dependence of the system, such crashes were becoming more likely [7]. Information security designers might wish to pay attention to the same concerns.

In the working group discussions at Connections 2018, Bond suggested developing a catalog of actions and likely effects to support more rigorous and evidence-based adjudication. The group also noted the security-effectiveness tradeoff as a suitable subject for unclassified games that would raise decision-makers’ awareness – some seem reluctant to concede that there even is a tradeoff. The U.S. Central Intelligence Agency is intensely examining this issue, trying to move toward greater transparency and cooperation with other agencies [8].

The development of metrics of cyber activity, effects and risks has also been somewhat neglected, with potentially serious consequences [9, 10]. A National Science Foundation-supported study at The Ohio State University, working with the Army Cyber Command, demonstrated a few classroom games that addressed serious policy issues without requiring classified information or deep technical knowledge by the players [11].

Conclusions

Wargaming may offer considerable potential to identify key issues and policy options in information security. Wargaming is better suited to illuminating decision-making than to assessing technical capabilities and possibilities. Engineering analysis, ideally including penetration testing, is more appropriate as the basis for risk assessments and adjudication.

It is possible to design and execute wargames of critical policy and practice decisions at a low level of classification. One particularly promising area is the tradeoff between too little security, resulting in catastrophe, and too much security, resulting in paralytic failure. Another is whether more complex and sophisticated, software-dependent security systems increase the chances for design and coding failures that cause the same kinds of damage as deliberate malicious attacks. While these are more technical than decision-making questions, wargaming them could focus attention on design issues that engineers expert in the technical details might tend to assume away.

There are many cautions, however. Wargaming technical issues can lead to useless conclusions and undermine the credibility of wargaming in general. For many applications, technically sound adjudication may take much longer than the time available for the game. On the other hand, some problems require far more engineering analysis than time and circumstances permit, in which case wargaming can at least alert decision-makers to the areas of greatest potential impact where more analysis is most urgently indicated.

Wargaming the decision-making questions well more strongly resembles economic, business and political games than traditional military conflict games. Building up a basis for better in-stride adjudication of such games could be of great benefit, but it will be a serious challenge. Wargaming is best viewed as an important part of a research cycle that also includes qualitative elicitation and problem definition, engineering, risk assessment and interpretation.

Writing this article and responding to sources’ comments reiterated an old lesson from journalism, which clearly applies to wargaming and related analyses as well: always, always rely on more than one source. Contrasting views are more often complementary than adversarial. Assembling the differing views into a more or less coherent picture is an extremely valuable skill.

OR/MS analysts fluent in both engineering and wargaming could find great opportunities in this rapidly expanding area of activity, with major benefits for the United States, its allies and trading partners, and numerous businesses. Both the Military Operations Research Society (MORS) and the Connections conferences, among other professional groups, offer many opportunities to learn more.

References

1. M. Robinson and S. Downes-Martin, 2018, “In-Stride Adjudication,” Working Group Report, Proceedings of the Connections US Wargaming Conference 2018, https://paxsims.wordpress.com/2018/09/12/in-stride-adjudication-connections-2018-working-group-report/.
2. Game Lab report, 2018, “How can we credibly wargame Cyber at an unclassified or low level of classification?” (S. Downes-Martin, chair), https://paxsims.files.wordpress.com/2018/08/game-lab-unclassified-cyber-gaming-20180814.pdf.
3. Game Lab report, 2018, “How can we design seminar wargames that avoid the risky shift and the honesty shift that occurs in small group discussions?” (S. Downes-Martin, chair) https://paxsims.files.wordpress.com/2018/08/game-lab-risky-and-dishonesty-shifts-in-wargames-20180812.pdf.
4. P. Perla and E. McGrady, 2009, “Systems Thinking and wargaming,” CNA report CRM D0020990.A2/Final November 2009.
5. “Propaganda,” https://www.physics.smu.edu/pseudo/Propaganda/.
6. “Institute for Propaganda Analysis records,” http://archives.nypl.org/mss/1513.
7. C. Lazzareschi, 1990, “What Happened When Nationwide System Crashed,” Los Angeles Times, Jan. 17, http://articles.latimes.com/1990-01-17/business/fi-212_1_reservation-system4.
8. D. A. Samuelson, 2017, “The CIA’s New Transparency: Analytics Plays a Role in Opening Access,” Analytics, January-February.
9. D. A. Samuelson, 2016, “Reducing Cybersecurity Risk: Better Metrics and Measurement Are the Key,” Analytics, September-October.
10. D. W. Hubbard and R. Seiersen, 2016, “How to Measure Anything in Cybersecurity,” Wiley.
11. D. A. Samuelson, 2017, “Wargames Illuminate Cyber Threat: Classroom games provide an analytic, educational toolset to analyze the effects of proposed courses of action,” OR/MS Today, August (with Olivia Kay Hernandez, Theodore T. Allen).