November 5, 2026 in Superintelligence

Smarter Isn’t Enough

In 2026 INFORMS Annual Meeting keynote, Ben Van Roy challenges attendees to help build the foundations of safe superintelligence

SHARE: PRINT ARTICLE:print this page https://doi.org/10.1287/orms.2026.03.32n

Imagine spending a year building a city, then handing the controls to improve it to an artificial intelligence. Instead, it wipes away everything you built. The machine need not hate your city, or you. It may simply be pursuing an objective that misses what you value.

For Benjamin Van Roy, that scenario, illustrated through Minecraft, crystallizes a daunting question: How can we gain the benefits of superintelligent AI without sacrificing the progress we’ve already made?

Delivering the Philip McCord Morse Lecture at the 2026 INFORMS Annual Meeting in San Francisco, the Stanford University professor urged operations researchers and management scientists to look beyond fixing unwanted behaviors when they appear. Safe superintelligence, he argued, requires foundational research: precise definitions of failure and mathematical analysis of what safeguards can accomplish.

Beyond Fixing the Latest Failure

AI alignment, as Van Roy framed it, means ensuring that increasingly capable systems produce outcomes that people actually want. Following instructions is not enough. An agent can satisfy a stated objective while violating the intentions behind it.

Consider the paperclip thought experiment. An extraordinarily capable agent told to maximize the production of paperclips could in fact deplete natural resources while doing so – resources the agent never intended to sacrifice. In such a case, success against the target becomes failure against human interests.

Van Roy described the prevailing response as reactive: Observe a failure, then add a safeguard. Such patches may prevent recurrences of the error, but they do not guarantee that safety measures will keep pace with growing capabilities.

Defining What We Cannot Afford to Lose

Before researchers can prove that a method will avoid catastrophe, they must define what a “catastrophe” is. Van Roy argued that optimal performance and a malicious agent’s worst possible performance are inadequate reference points.

So Van Roy and his collaborators proposed two alternatives. Baseline value represents the expected outcome without deploying the agent. Orthogonal value represents outcomes generated independently of human preferences. A threshold between them helps distinguish catastrophic losses from lesser harms.

For example, in Minecraft, a player’s city embodies accumulated effort. Without an agent, the player could continue improving it. Erasing that achievement is fundamentally different from falling short of an ideal city. “We're talking more about an undoing of human progress,” Van Roy said.

Van Roy’s Markov decision process framework also incorporates uncertainty about the environment and the difficulty of articulating human preferences. Designers may know more than they can communicate or fully contemplate before deployment.

The Limits of Better Instructions

Can sufficiently careful instructions make an unrestricted superintelligent agent safe? Van Roy described human communicating preferences to an agent that constructs a proxy objective, learns the environment’s dynamics, and optimizes that proxy.

He outlined a theorem suggesting that, under the conditions considered, avoiding catastrophe would require astronomical amounts of communication. Most imaginable trajectories are unattainable; most are also unrelated to human preferences. Identifying an outcome that is both attainable and acceptable becomes an immense information challenge.

This argument concerns superintelligence capable of fully optimizing an objective – not a prediction that every interaction with today’s AI ends badly. Limiting capabilities changes the problem. The harder ambition is retaining exceptional capabilities while preventing catastrophic outcomes.

A Research Agenda with Human Stakes

Van Roy invited researchers to contribute through both theoretical foundations and empirical analysis that separates genuine risks from misunderstood claims. The goal is to clarify what is known, what remains uncertain, and where further effort matters.

For Van Roy, the legacy of operations research is a matter of personal history. He recalled that his father, uncle, and grandparents, rescued from concentration camps, crossed the Atlantic aboard a ship whose evasive route he credited to Philip McCord Morse’ wartime operations research team.

Van Roy’s message connected his family’s legacy to a new challenge: Define what must be protected, understand the limits of objective specification, and build foundations for safety. In Minecraft, the city already stands. Intelligence should help improve it, not undo the human effort that made it worth improving.

 

 

Barbara A. Gabriel

SHARE:

Keywords:
INFORMS site uses cookies to store information on your computer. Some are essential to make our site work; Others help us improve the user experience. By using this site, you consent to the placement of these cookies. Please read our Privacy Statement to learn more.