Alignment of Risk Attitudes in AI Risk Management Systems: Four Examples in Medicine, Defense, Sports, and Autonomous Vehicles

Some artificial intelligence (AI) systems based on large language models are designed to provide risk management decisions. These AI decisions are generally made under uncertainty and involve an encoded risk attitude. Whether that risk attitude fits the situation or not is illustrated here by four real-life examples, where the decision makers may choose to adopt or ignore the AI message. These examples include a medical test decision, a military case about the use of AI in the operation of autonomous drones in combat, AI guidance in competitive sports with an application to sailing races, and the management of autonomous vehicles. The potential users of the information may or may not realize the existence and the implications of the encoded risk attitude. If they do and they do not agree with it, they may ignore the recommendations of the AI system. Therefore, for the message to be useful, the AI risk attitude must be understood, and it must fit the preferences of the decision maker. The objective of this paper is to illustrate and compare the effects of these potential conflicts of values in the real world through those real-life examples. It emphasizes the benefits of decision makers’ education about risk attitudes, and the need to align the AI system whenever feasible.

History: This paper has been accepted for the Decision Analysis Special Issue on the Implications of Advances in Artificial Intelligence for Decision Analysis.