Why AI Adoption Needs a Framework of Trust

The artificial intelligence (AI) industry is quickly evolving as AI technology continues to be tasked with solving myriad problems across enterprises. Today, AI holds enormous potential to revolutionize business decisions, but AI is not a one-size-fits-all, hands-off guarantee of accuracy and optimization. Utilizing AI to solve problems is not without risk, so users need to think carefully about how much to trust AI and investigate how reliable the generated results actually are. Likewise, organizations must consider different levels of human intervention in AI to monitor its efficacy, accuracy and progress. More importantly, they should examine, and agree on, how to measure the overall “trust” of the AI system itself.

Allowing AI to make unchecked decisions can lead to bad business strategies that seriously impact customer loyalty and profits. Even worse, an unfettered AI can make decisions considered immoral or illegal. The industry continues to grapple with these ethical considerations such as how to keep racial bias out of AI in policing systems or banking decisions.

So, do these risks mean that a human must always supervise an AI system? Not necessarily. There are challenges and threats inherent to most decision-making processes, and AI is no different. The key is to assess the level of risk and take steps to build a strong, trusted solution similar to how you train and build trust in an employee’s ability to make daily decisions on their own.

Understanding When to Monitor AI to Mitigate Risk

To determine when it’s appropriate to be laidback with your AI, there are two basic dimensions of risk to consider.

• How catastrophic would it be if the AI delivered the wrong choice?
  Start by considering how damaging it would be if the AI fails. For instance, if an AI is designed to find targets for a weapons system, it is imperative for the system to be accurate because the consequence to human wellbeing is extremely high if the AI fails. On the other hand, if an AI is tasked with writing a limerick, there’s not much consequence if it fails to deliver in using the appropriate structure.
• How difficult is it for the AI to make the correct choice?
  The next question to ask is how difficult is it for the AI to get it right in the first place? Some problems are fairly well-solved and others are not. For example, in the case of exchanging money between banks, there is a sophisticated software language that exists to execute these transactions between machines, meaning an AI can likely carry out a task like this with a high degree of accuracy.
  But not all problems are solved this simply. Predicting human behavior and incorporating many shifting parameters and unknowns is a complicated task, and one that AI doesn’t necessarily get right all the time. With hard problems like these, accuracy is not guaranteed.

Problems that are easy and of low consequence are great candidates for hands-off, unmonitored AI. But when there is a high uncertainty of success and serious consequences for failure, it is imperative to assess and build trust in an AI. For instance, let’s turn to the self-driving car industry to discuss a framework for characterizing trust.

What We Can Learn from the Self-driving Car Industry

Engineering an AI to drive cars safely is a difficult technical problem and one in which failure is at a high consequence of physical injury. With a problem like this, it’s imperative to have an AI that you trust completely to handle any and all scenarios that can come up while driving.

The self-driving car industry has developed a framework for measuring trust in AI by describing them with specific levels of automation. Levels of automation, which usually range from 1 to 5, are a way to characterize how much human intervention is needed for an AI to operate as desired.

Level 1 can be thought of as driver assistance. A good example of this is cruise control. Here, the software ensures the car is driving at a certain speed, but the driver themselves is responsible for steering and braking. At Level 2, the AI begins to assume some of the ability to steer and accelerate, but the driver is still monitoring and responding to the environment. Other examples of AI at these levels include systems that detect when the car is drifting over the lane line. For example, in Level 1, the AI can alert the driver that there may be an issue they need to pay attention to, but it does not take any action itself to control the car. In Level 2, the AI might include an automatic braking system when the car gets too close to another car.

At Level 3 of automation, the AI is able to drive the car in perfect conditions. This can be thought of as “conditional automation.” Level 4 AI, such as those with “high automation,” can drive the car in more diverse conditions, but still, crucially, with the option of driver override.

At Level 5 or “full automation,” the AI truly takes over all operation of the vehicle and can handle all unusual cases. The driver can sit back fully and not pay attention. Because AI failure in this scenario could be catastrophic for the driver, they need to have full trust that the AI is truly at a Level 5.

Realistically speaking, not every AI needs to get to Level 5 automation. Having Level 1 assistance when driving a car is still quite beneficial to the driver. But what’s important is the concept that not all AI can provide the same level of automation, and the distinction between Level 2 and Level 5 is critical.

Determining When AI is Ready

Human decision-making is critical to setting the basic parameters of what you want your AI to do, or not do. Every AI needs a human to tell it what is appropriate and make value judgments before it is sent off to solve problems. Once the AI is up and running, a human check on the AI’s output depends on how risk is characterized. Assessing risk is the first step before deciding how much human intervention is needed. Levels of automation are another good way to characterize how much to trust the AI. Ask yourself: Is your trust high enough to mitigate the risks? This framework or mentality is similar to what should be applied to any problem that the business is hoping to solve by automating a process.

For example, imagine you’re developing a new marketing strategy to better engage your customers. The risks of a bad strategy, such as sending the wrong promotions, are high because they run the risk of losing money and/or alienating customers. Marketing teams that don’t use AI also grapple with this.

If the business wants to use AI to develop that strategy, they need to decide whether it needs to be fully automated in the first place, or if humans should provide a final check-in to maximize success. If the decision is to go hands-off, the next question to ask is whether the AI is strong enough to factor in all edge cases. After all, marketing problems incorporate lots of data about millions of customers’ behavior; therefore, it might make sense to have a final human check before implementing any AI recommendations. If opting to forgo that final check, it’s imperative that the AI is operating at a Level 5 capability.

There’s tremendous value in all levels of AI, such as Level 1 AI that alerts a driver to another car in a blind spot. But some problems require much more trust. If you want a fully self-driving car, you need a system at Level 5. However, every level of AI has the potential to be useful in diverse scenarios. What really matters is that every application of AI is carefully thought through and the level required to mitigate the risks associated with the task at hand is agreed upon in advance.