When Does Artificial Intelligence Trigger Employees to “Game the System”?

Key Takeaways

• Delegating decisions without understanding how employees behave when interacting with AI prevents top management from ensuring that employees use AI in line with the goal for which AI was initially adopted.
• The complexity of AI in the workplace can result in unintended, and often harmful, human behaviors that not only negatively impact organizational performance but can also have wider societal consequences.
• The distinctive capability of AI to track and analyze employee behavior can be strategically exploited by employees, who may use their “local knowledge” to experiment with how AI can enhance their performance with minimal effort.

Policymakers are increasingly concerned about the potential harm artificial intelligence (AI) could cause to employees’ work lives within organizations. However, recent trends show that employees often “game the organization,” leading to possible negative societal consequences. A prominent example is Uber drivers, who have gamed the system by coordinating their login and logoff times to trigger higher surge prices. This coordinated behavior, a form of moral hazard, is just one example of how employees strategically coordinate their behavior to maximize personal or team gain based on what the algorithm rewards. The mechanism is rather simple: Because the algorithm is often opaque, employees focus their gaming strategy on influencing the data collected about them.

The increasingly complex and crucial question facing organizations is: When and how does AI lead to unintended human behaviors? This article argues that delegating decisions without understanding how employees behave when interacting with AI prevents top management from ensuring that employees use AI in line with the goal for which AI was initially adopted. In practice, AI is often too complex to be fully understood by managers, who often lack an a priori knowledge of both the potential performance improvements AI can offer and the behavioral response AI may trigger in employees. This uncertainty can result in unintended, and often harmful, human behaviors that not only negatively impact organizational performance but can also have wider societal consequences.

The theme is timely, as human-AI interaction ranks high on the agenda of organizations and policymakers yet remains poorly understood in many aspects. This article proposes solutions to mitigate the risks of unintended employee behaviors in the form of coordinated moral hazard, thus optimizing human-AI collaboration, reducing uncertainty around AI and maximizing its value for organizations.

Let me illustrate this problem with an example: The adoption of AI in an organization aims to enhance productivity and overall performance. We can think of the organization’s performance as the “weighted sum” of the performances of its “modules” (e.g., teams, divisions, departments). Assume that AI is shared across these modules at no cost, meaning that there is no competition for its use. Moreover, there is no time limitation on its use, so all modules can use AI simultaneously. This scenario applies to many AI applications in organizations (see, for instance, the “AI around the world” project from the Deloitte Center for Government Insights). 

What is the core issue here? The challenge is that the CEO (or another top manager responsible for AI adoption) faces uncertainty about the expected performance improvements after the AI adoption. Will the organization’s performance increase by 20%? 100%? Or could it decrease in the short term? Often, the CEO lacks precise priors, beliefs, expectations or reference points about how and how much AI will impact the performance of each module. For example, the way AI interacts with each module’s operations may vary, but the CEO cannot predict in advance how each module will use the AI or to what extent it will enhance their performance.

In theory, AI can help organizations reduce agency costs that arise from information asymmetry between the modules and top management. By tracking and analyzing employee behavior, AI enables CEOs to gain insights into past actions and anticipate future behavior, ensuring alignment with organizational goals. However, in practice, the distinctive capability of AI to process vast amounts of data can be strategically exploited by the modules’ employees. Being the direct users of AI, employees may use their “local knowledge” to experiment with how AI can enhance their module’s performance with minimal effort – especially in mechanistic workplaces in which it is easier for employees to estimate “how much AI” will speed up the execution of their tasks. Thus, AI could unintentionally incentivize employees to engage in “gaming the system” behaviors.

In essence, AI introduces a layer of uncertainty between the modules using it and top management. Employees may exploit this uncertainty, coordinating their effort reduction in a way that benefits their module while still appearing productive. Although AI may improve accountability by reducing the agency costs faced by the CEO, when AI is shared among competing organizational modules, it can lead to coordinated reductions in effort, undermining overall organizational performance.

What could be the organizational response to this issue? First, top management could align employee well-being with the original goal of maximizing organizational performance. This approach accommodates the reduced effort of modules while introducing training activities to reskill and upskill employees. For example, part of employees’ contracted hours could be dedicated to training to learn how to effectively interact with AI. This solution allows employees to maintain their wages while working less hours and simultaneously learning to productively collaborate with AI. This solution fosters organizational trust, builds organization-specific human capital and promotes efficient human-AI collaboration, which are critical components of an organization’s competitive advantage.

Second, when precise knowledge of the impact of AI on employee performance is unavailable, the organization should introduce competition among modules for access to AI resources. Without competition for this “intelligent resource,” the organization’s modules may exploit top management’s information gap, leading to collusion and reduced effort. As a result, while the performance of the modules may improve compared with pre-AI levels, it will fall short of the potential performance – unknown to management. To address this, a bonus scheme based on relative performance between modules should be implemented. Because top management cannot directly measure the effort of each module, a bonus system that rewards relative performance encourages modules to avoid collusion and minimize effort reductions. This scheme incentivizes competition and aligns module performance with organizational goals.