OptiChat: Bridging Optimization Models and Practitioners with Large Language Models

1. Introduction

Significant progress in large language models (LLMs) has been witnessed in recent years (OpenAI 2023, Touvron et al. 2023), with applications emerging in chemistry (Boiko et al. 2023, Li et al. 2023b, Bran et al. 2024, Jablonka et al. 2024), biology (Luo et al. 2022, Lin et al. 2023), healthcare (Sallam 2023, Thawkar et al. 2025), finance (Dowling and Lucey 2023, Wu et al. 2023b, Lopez-Lira and Tang 2024), manufacturing (Badini et al. 2023, Wang et al. 2023), supply chain management (Li et al. 2023a), and optimization modeling (AhmadiTeshnizi et al. 2023, 2025; Ramamonjison et al. 2023; Ahmaditeshnizi et al. 2024; Xiao et al. 2024; Tang et al. 2025). This widespread applicability highlights the capability of LLMs to comprehend and process natural language across diverse domains. Leveraging the versatility of LLMs, a recent study introduced an interactive dialogue system, TalktoModel (Slack et al. 2023), designed to assist practitioners in understanding complex machine learning models from various application domains by integrating LLMs with explainable artificial intelligence (XAI) techniques (Ribeiro et al. 2016, Lundberg and Lee 2017, Selvaraju et al. 2017, Karimi et al. 2020). Case studies demonstrated that the interactive dialogue system of TalktoModel significantly enhanced healthcare workers’ understanding of a machine learning-based disease prediction model.

In addition to machine learning models, optimization models are another type of complex model widely used across various fields, such as engineering, economics, healthcare, and manufacturing (Rardin 2016). These models formulate real-world decision-making problems into optimizing an objective under a set of constraints (e.g., budgets and operational limits), where each decision variable has a well-defined physical meaning. For example, a supply chain optimization model aims at determining the most economically efficient production levels and product distribution while meeting customer demands. Although optimization models are built on structured and interpretable mathematical formulations that are accessible to experts, practitioners face significant challenges in understanding the abstract formulations and often perceive them as black boxes. This challenge becomes even more pronounced when the solutions suggested by these models deviate from familiar heuristics, leaving practitioners uncertain whether to trust them. To address these challenges, the optimization community has also developed explanation techniques analogous to those in XAI. Since the 1980s, the community has developed expert systems (Greenberg 1983, 1987), interpretable decision rules (Bertsimas and Stellato 2021, Goerigk and Hartisch 2023, Lumbreras et al. 2024), argumentation-based methods (Collins et al. 2019; Čyras et al. 2019, 2021), and counterfactual explanations (Forel et al. 2023).

However, the effective use of these explanation techniques still requires substantial optimization expertise. It can still be challenging for the end users of the optimization models, such as logistics coordinators, to independently comprehend the model’s outcomes, reconcile results that conflict with their own experiences, or conduct further analysis. This limitation imposes a significant burden on the optimization experts tasked with communicating the results to practitioners, slows decision-making processes, and prohibits the dissemination of optimization models in areas where optimization experts are inaccessible.

To address this limitation, a natural approach is to leverage LLMs for explaining optimization models given their versatility in natural language-based tasks. However, the hallucination of existing LLMs is concerning because it can produce unfounded or inaccurate explanations (Ji et al. 2023). One solution proposed by Li et al. (2023a) is to rely entirely on code generation to handle all queries for a supply chain optimization model. Alternatively, our previous work (Chen et al. 2024) augments LLMs with predefined functions, but this system is specifically designed for diagnosing infeasible optimization models. Moreover, there is currently no general-purpose platform tailored for explaining optimization models using natural language, nor are there open-source data sets available for systematically evaluating such systems.

In view of these challenges, we introduce OptiChat, an LLM-assisted system designed to not only interpret optimization models in natural language but also, provide post hoc explanations through interactive dialogues after the optimization model is solved. Common types of user queries can be classified and systematically addressed by OptiChat. To achieve this, we combine various predefined functions and code generation to draw faithful and trustworthy conclusions from optimization models. We also curate a comprehensive data set to evaluate OptiChat, supporting further research in this area.

As illustrated in Figure 1, the input to OptiChat is a well-written code script corresponding to an optimization model in the Pyomo/Python (Hart et al. 2011) algebraic modeling language developed by an optimization expert. OptiChat initially processes the code to generate a coherent and easy-to-understand description to assist practitioners in understanding the problem context. After that, practitioners can ask queries of interest in an interactive dialogue. We survey the queries gathered from practitioners and classify them into five categories: diagnosing, retrieval, sensitivity, what-if, and why-not queries (see illustrative examples in Figure 2). Diagnosing queries focus on fixing infeasibility issues. Retrieval queries seek to extract relevant model data. Sensitivity queries measure the impact of parameter uncertainties on the optimal objective value. What-if queries evaluate significant changes in input parameters specified by users. Lastly, why-not queries investigate the counterfactual scenarios suggested by users. These are termed as solution-specific queries, meaning that OptiChat must derive a solution from the model to provide an accurate explanation. Underneath, explanation strategies for each query category are either implemented as predefined functions or generated as code by the LLM, and they are then executed using an optimization solver, such as Gurobi. These two approaches are tailored to achieve high correctness rates in answering queries. Besides the solution-specific queries, the users can also seek qualitative explanations from the LLM. These queries, termed as solution-agnostic queries, are contextualized within the LLM’s memory using the input optimization model and the chat history to ensure the quality of the answers.

2. Related Works

We give a brief review of recent applications of large language models in operations research (OR). We divide the work into three different categories, which correspond to the typical workflow of developing and adopting an optimization model. (1) Use an LLM to formulate optimization models based on the problem statement. (2) Use an LLM to help develop a new optimization algorithm. (3) Use an LLM to facilitate interaction with the user. Our work belongs to the third category.

2.1. Formulating Optimization Models

OptiMUS (AhmadiTeshnizi et al. 2023, 2025) is a multiagent LLM-based system designed to formulate and solve linear programming (LP) and mixed-integer linear programming (MILP) problems from natural language descriptions. The system can develop mathematical models, write and debug solver code, evaluate generated solutions, and iteratively improve model and code efficiency and correctness based on these evaluations. Huang et al. (2025) proposed the operations research language model, a novel approach to OR modeling that relies on fine-tuning a semisynthetic data set rather than using prompt engineering or agentic models. Xiao et al. (2024) introduced a multiagent cooperative framework called chain of experts for automated OR problem modeling. Astorga et al. (2024) developed a Monte Carlo tree search (MCTS)-based approach that decomposes the modeling process into sequential stages (e.g., variables, objectives, and constraints) and uses MCTS to explore plausible model structures. Yang et al. (2024) presented OptiBench, a benchmark suite for end-to-end optimization problem-solving with human-readable inputs and outputs.

2.2. Develop Novel Optimization Algorithms

Another line of work focuses on the development of novel OR algorithms. FunSearch (Romera-Paredes et al. 2024), which explores the function space, is an evolutionary procedure that pairs a pretrained large language model with a systematic evaluator. It has been successfully applied to enhance online bin-packing heuristics by evolving the heuristics generated by the LLM. A genetic programming algorithm is employed to balance exploitation and exploration within the database of LLM-generated programs. Building on the pioneering efforts of FunSearch, several subsequent works have extended similar methodologies to other combinatorial optimization problems (van Stein et al. 2025).

2.3. Facilitating User Interactions with Optimization Models

The final research direction focuses on using LLMs to ease user interactions with optimization models. Lawless et al. (2024) demonstrated the use of LLMs to allow users to customize meeting preferences in a constraint programming-based scheduling model. Ju et al. (2024) designed an LLM-based system for travel planning. Kikuta et al. (2024) applied LLMs to explain solutions to vehicle routing problems. Closest to our framework is OptiGuide (Li et al. 2023a), which is a multiagent system for explaining supply chain models by relying entirely on LLM-based code generation to answer user queries. Our main innovation lies in using predefined functions to improve the accuracy and speed of the agentic framework as demonstrated through our ablation studies. Compared with the agentic framework of Li et al. (2023a), OptiChat has syntax reminders and operator agents to coordinate the execution of the predefined functions. We also develop a data set to evaluate our proposed framework on applications across multiple domains rather than being restricted to a single-use case. This work builds upon our earlier system (Chen et al. 2024) for diagnosing infeasibility, which was not agentic and did not support queries related to optimality.

3. Methods

OptiChat is composed of a user interface, a structured sequence of LLM-based agents, an optimization solver, a modeling software, and several custom-built functionalities. In this work, OptiChat utilizes GPT-4 (OpenAI 2023) for the LLM-based agents, Gurobi (Gurobi Optimization LLC 2022) as the optimization solver, and Pyomo (Hart et al. 2011) as the algebraic modeling language. In this section, we first motivate the functionalities of OptiChat by providing background on common problems faced by practitioners and connecting them with the proposed explanation strategies. We then present the design of our multiagent framework, outlining the role and subtask of each agent. Lastly, the exception management for specific queries is discussed. The implementation of OptiChat is available on GitHub: https://github.com/li-group/OptiChat.git.

3.1. Problem Statement and Explanation Strategies

3.1.1. Feasible/Infeasible Model Description.

Most practical applications in optimization can be formulated as MILPs/LPs:

where decisions to make are denoted as $\mathbf{x}$, which can consist of both integer and continuous variables; $\mathbf{c}$ represents the cost coefficients. The problem is subject to linear constraints $\mathbf{A}\mathbf{x}\le \mathbf{b}$: $\mathbf{A}\in {\mathbb{R}}^{m\times n}$, $\mathbf{b}\in {\mathbb{R}}^m$. However, practitioners often struggle to understand the mathematical formulation. More importantly, optimization models derived from real-world problems can be infeasible at the initial stage. Infeasibility does not refer to code bugs that prohibit models from being executed but rather, to an optimization model in which no solution exists to satisfy all constraints simultaneously. This usually occurs because of overly restrictive constraints and inaccuracies in the underlying data. For example, in an aggressive overselling scenario, an aircraft may lack sufficient seats to accommodate all ticketed passengers. This infeasibility further complicates practitioners’ ability to identify flaws in the model.

An irreducible infeasible subset (IIS) is a minimal set of constraints and/or variable bounds within an optimization model that causes infeasibility defined by two key properties. (i) The IIS itself is infeasible, and (ii) any proper subset of the IIS is feasible. In other words, we can use the IIS to extract the components in violation and characterize the nature of infeasibility. Different algorithms have been developed to isolate IIS, such as the deletion filter (Chinneck and Dravnieks 1991), additive method (Tamiz et al. 1996), and hybrid approach (Guieu and Chinneck 1999). Commercial optimization solvers, like CPLEX (IBM 2022) and Gurobi (Gurobi Optimization LLC 2022), have implemented variants of these IIS detection algorithms. For a comprehensive review of IIS detection, readers are referred to the monograph (Chinneck 2008).

3.1.2. Diagnosing Query.

Restoring feasibility can be approached in various ways, but not all are actionable in the real world. To restore feasibility, practitioners seek strategies that align with their operational priorities. For instance, suppose a model is infeasible to produce a chemical at a 99% concentration with the current resources available in a chemical plant. Increasing the conversion rate of an existing chemical reaction is generally impractical, whereas negotiating with business partners to lower product purity requirements is more actionable.

To address this need, either (1) one can recursively remove constraints from the IIS until the model becomes feasible, or (2) one can add slack variables to the optimization problem and allow for adjustments to input parameters. The second approach is more practical as each constraint reflects an important aspect of the problem and cannot always be removed in practice. In contrast, introducing slack variables into the optimization problem in (1) provides practitioners with a more concrete and actionable plan for feasibility restoration. Mathematically, the following extended problem is solved:

where $\mathit{\delta }{\mathbf{A}}^{+}$, $\mathit{\delta }{\mathbf{A}}^{-}$, and $\mathit{\delta }{\mathbf{b}}^{+}$ are nonnegative slack variables. Because all of the constraints are inequalities, adding nonnegative slacks $\mathit{\delta }{\mathbf{b}}^{+}$ on the right-hand side relaxes the problem. In contrast, adding the left-hand-side slacks, $\mathit{\delta }{\mathbf{A}}^{+}$ and $\mathit{\delta }{\mathbf{A}}^{-}$, is less desirable. It is worth noting that because only a subset of parameters can be adjusted in practice, the slack variables differ in dimensionality from the parameters $\mathbf{A}$ and $\mathbf{b}$. The objective is to minimize the total perturbation to the original problem by summing up all of the slack variables. In principle, different weights could be assigned to different slack variables, representing the cost of perturbing the corresponding parameters. The optimal solution found in (2) can be explained as the minimal adjustments needed to restore feasibility.

It should be noted that adding slack variables to the left-hand-side parameters $\mathbf{A}$ will lead to a product of variables between $\mathit{\delta }{\mathbf{A}}^{+}$, $\mathit{\delta }{\mathbf{A}}^{-}$, and $\mathbf{x}$ in (2). This results in a nonconvex mixed-integer quadratically constrained program (MIQCP), which is often prohibitive to solve. In many situations, left-hand-side parameters represent immutable properties. For example, in the constraint $\mathbf{A}\mathbf{x}\le \mathbf{b}$, where $\mathbf{b}$ is a deadline, $\mathbf{x}$ indicates task status, and $\mathbf{A}$ represents processing times. The processing times are an inherent property of the machines and cannot be changed. When OptiChat detects a request to add slacks to $\mathbf{A}$, it will alert users of this immutability before initiating the MIQCP.

3.1.3. Retrieval Query.

An optimization model consists of decision variables, parameter data, constraints, and an objective function. Optimization models developed for practical use are often large in scale, with some components indexed over large sets. For example, a variable that represents the assignment decision of an aircraft can be indexed over hundreds of routes. When practitioners need to review specific data or optimal decisions from the model, it is inefficient to retrieve this information. OptiChat facilitates real-time access to specific model information through natural language.

3.1.4. Sensitivity Query.

Many optimization models are constructed with incomplete knowledge of problem parameters. For example, electricity prices and customer demands often fluctuate in the market over time and cannot be fixed within the model. The values of these parameters can be predicted using historical data and updated on a rolling basis. It is important for practitioners to evaluate how changes in problem parameters impact the optimal objective value, undertake risk assessment, and devise appropriate management strategies.

We propose to perform sensitivity analysis based on well-established duality theory in linear programming (Bertsimas and Tsitsiklis 1997), where $\mathbf{x}$ does not contain integer variables. In short, the change in the optimal objective value in response to changes in input parameters can be expressed as a value function:

where $\mathcal{X}=\{\mathbf{x}:\mathbf{A}\mathbf{x}\le \mathbf{b}\}$ and ${\mathbf{y}}^{*}{}^{\top }$ is the optimal solution to the dual problem of (1). Problem (1) is called a primal problem that finds the minimum cost while satisfying all of the constraints, whereas the dual problem of (1) aims to seek the tightest lower bound on the optimal primal cost. Denote the original data given in the primal problem as ${\mathbf{b}}^{*}$; Expression (3) indicates that when the perturbed parameters $\mathbf{b}$ are close to ${\mathbf{b}}^{*}$, the optimal primal objective value is a linear function of the input parameters with gradient ${\mathbf{y}}^{*}$. In other words, it precisely reflects the sensitivity of the optimal objective value in the vicinity of ${\mathbf{b}}^{*}$. The access to dual information in LP is supported by most optimization solvers (Gurobi Optimization LLC 2022, IBM 2022, MOSEK ApS 2023).

3.1.5. What-if Query.

Although sensitivity analysis can evaluate local changes in parameters, it is not a valid methodology to evaluate larger changes. This limitation arises because the optimal dual solution in (3) is not constant but depends on parameters. After large parameter perturbations, the dual solution previously found in the original problem no longer accurately reflects the change in optimal objective value. These significant perturbations often occur when a new policy is being established or when the industry is evaluating a new business strategy. For example, practitioners may pose questions, such as “What if we increase the labor force to 35 people?” and “What if customer orders are cut by a third?”

In these cases, OptiChat systematically identifies the extent and type of modification from the user’s query. Model (1) is then revised by updating parameters and constraints accordingly:

In this revision, $\widehat{\mathbf{c}}$, $\widehat{\mathbf{A}}$, and $\widehat{\mathbf{b}}$ are completed by the interaction between LLMs and our predefined function. Modifications will change the feasible region of the problem, generally leading to a different optimal solution.

3.1.6. Why-Not Query.

Mathematical optimization is a rigorous methodology that searches for the optimal solution. In contrast, practitioners often rely on their experience to make decisions. As a result, it can be challenging to convince business managers or stakeholders to accept the optimal solutions given by an optimization model, especially when these solutions are against their intuitions. For example, one might ask “Why not choose supplier 1?” in a supply chain model, even though the current optimal solution does not include it.

A counterfactual explanation addresses the “why-not” queries by examining alternative decisions or outcomes that are not currently realized in the optimal solution. To investigate such a counterfactual scenario, we modify the original model by incorporating an additional set of constraints that force the desired alternative to occur. In this example, if $x_1$ represents the binary decision to select supplier 1, we add the constraint $x_1=1$ to enforce this choice. Once we resolve the modified problem with this counterfactual constraint in place, we can observe how the objective value and the feasibility of the solution change. For instance, the forced selection of supplier 1 may raise production costs, reduce overall profitability, or even make the problem infeasible. By comparing the new solution with the original one, we gain insight into the trade-offs and underlying reasons that the original optimal solution did not include supplier 1. This counterfactual reasoning thus provides a transparent and actionable explanation of the model’s decision-making process, illustrating which model parameters and constraints are most influential in ruling out the queried alternative. The explanation for the “why-not” query relies on code generated by the LLM because the counterfactual can be any arbitrary fact suggested by the practitioners, for which predefined functions do not suffice.

3.2. Multiagent Framework

Figure 3 depicts the workflow of the multiagent framework. Before the interactive dialogue, OptiChat begins with the illustrator agent, which preprocesses the Pyomo optimization model uploaded by users. During the conversation, the coordinator routes queries either to the explainer agent when they can be directly answered by an LLM or to the team of engineering agents when more specialized reasoning is required. The engineering team operates in a structured order, comprising the following agents: (1) the operator who controls all predefined functions, (2) the programmer and evaluator agents who are responsible for generating code for tasks that cannot be addressed by the predefined functions, and (3) the reminder agent who provides supplementary information to enhance the accuracy of the other two agents. The prompts are shown in Appendix C. To illustrate the workflow, a what-if query is provided as an illustrative example in Appendix D.

3.2.1. Illustrator.

This agent extracts sets, parameters, variables, constraints, and objectives from the Pyomo optimization model, labeling each component with a natural language description. These descriptions establish a dictionary that maps the physical meanings of model components to their corresponding notations in the source code. Because users may refer to model components using different terms, this preprocessing enables LLMs to accurately identify the referenced component by providing the problem context. Alongside these descriptions, information about components, such as index dimensions and solution status, is also stored to support interaction with optimization tools. Finally, the illustrator agent describes the model to users in a concise and coherent manner using natural language. If the solution status of a model indicates infeasibility, the agent will also invoke tools to compute the IIS and provide troubleshooting recommendations by interpreting the conflicting constraints.

3.2.2. Coordinator.

After the illustrator agent preprocesses a model, users can submit queries to initiate an interactive conversation. The coordinator agent classifies each query as solution agnostic or solution specific. Solution-agnostic queries are solely addressed by the explainer agent through in-context reasoning, whereas solution-specific queries require technical feedback from agents on the engineering team before being forwarded to the explainer agent.

3.2.3. Explainer.

The explainer agent is responsible for conveying any technical information to practitioners in an understandable manner as the end point for all queries.

3.2.4. Reminder.

The reminder agent serves as the entry point for every solution-specific query, guiding LLMs to more accurately determine which function to invoke, which model component to modify, and which specific index within that component to reference. Among these tasks, selecting the correct component index (e.g., a parameter index) is the most error prone. In contrast, identifying the appropriate function name is generally easier for LLMs, aided by few-shot demonstrations. The model description provided by the illustrator agent helps the reminder to pinpoint the relevant component name. The syntax guidance offered by the reminder agent has been shown in our ablation studies (Section 4.2.4) to improve the accuracy in identifying the correct function, component names, and indices.

3.2.5. Operator.

As proposed in Section 3.1, we implement four predefined functions in the operator agent to address the diagnosing, retrieval, sensitivity, and what-if queries in Figure 2. Informed by the specific syntax guidance, this agent selects the appropriate function and arguments, and then, this agent invokes the corresponding tool to generate the solution.

3.2.6. Programmer and Evaluator.

The development of the programmer and evaluator agents is inspired by recent works on applying code generation to optimization (Li et al. 2023a, Ahmaditeshnizi et al. 2024). The code generation capability of LLMs has been widely explored and demonstrated impressive results across various tasks (Wu et al. 2023a). These agents are designed to address the why-not queries in Figure 2 but also potentially handle unexpected queries beyond the reach of predefined functions. The programmer and evaluator are involved in a loop with limited iterations. The evaluator first executes the code generated by the programmer and then, reviews the terminal outputs and error messages if available. Throughout this process, the code is automatically refined until a bug-free and comprehensive solution is produced. The prompts for why-not queries are designed to guide the LLM in generating additional constraints from the user’s counterfactual query, thereby narrowing the LLM’s task scope and reducing its code output.

3.3. Exceptions Management

When practitioners interact with OptiChat, the system is robust in handling exceptions caused by the lack of optimization expertise. The sensitivity analysis in OptiChat relies on the strong duality of LP. However, practitioners may request sensitivity analysis concerning left-hand-side parameters $\mathbf{A}$ in LP models or parameters in MILP models. Unlike the sensitivity of $\mathbf{b}$ in (3), the dependence of the optimal objective value on left-hand-side parameters $\mathbf{A}$ cannot be determined based on duality theory. When the optimization model is an MILP, the equality in (3) no longer holds, breaking the connection between the optimal objective value and $\mathbf{b}$. In this case, OptiChat will notify them that sensitivity analysis is not supported and suggest providing specific modifications for evaluation if they still wish to address the same queries. This converts a sensitivity query into a what-if query, which can be addressed in a less restrictive manner. Similarly, when OptiChat detects a request to add slacks to $\mathbf{A}$ in (2), it will issue a warning message to indicate the potential immutability of left-hand-side parameters $\mathbf{A}$ and the increased processing time required to initiate the MIQCP. The model information, such as whether a parameter appears on the left-hand side, is stored during the preprocessing step. This information will be automatically used to verify the presence of such exceptions in the predefined functions and guide the user accordingly. Furthermore, when unexpected failures occur in predefined functions, the programmer and evaluator agents will be invoked to generate code-based solutions.

4. Results and Discussion

In this section, the effectiveness of OptiChat is demonstrated in terms of model descriptions and query responses. First, we emphasize the efficiency of OptiChat in generating autonomous model descriptions compared with consulting with optimization experts. In terms of the interactive dialogue, we measure the correctness rates of responses for each query type, providing a quantitative evaluation of OptiChat’s accuracy. We also present a qualitative showcase of query responses, illustrating how OptiChat assists practitioners in real-world settings.

We test OptiChat on 24 optimization models written in the Pyomo/Python framework. To demonstrate the versatility of LLMs, the 24 models span various contexts, including supply chain, manufacturing and production, petroleum refinery, industrial scheduling, chemical and process system engineering, transportation, etc. The 24 models are adapted from the General Algebraic Modeling System (GAMS) Model Library¹ sourced from the Pyomo Cookbook by the University of Notre Dame² and a public GitHub tutorial³ or adapted from an optimization textbook (Rardin 2016). Infeasible variants of these models are created by adjusting the model parameters or introducing additional constraints. A summary of the statistics, including the numbers of variables, constraints, and parameters, is shown in Appendix A. The size of the models is orders of magnitude larger than those in existing natural language to OR models benchmarks.

OptiChat aims to make optimization models more accessible to a wider audience by enabling seamless interaction between users and the underlying models, thereby reducing the time that experts must spend communicating with users. For our experiments, we recruited 29 experts, including graduate students and postdoctoral researchers, each with at least one year of experience in optimization theory or modeling. These experts were tasked with drafting model descriptions and answering follow-up queries.

4.1. Model Description

In this study, the expert participants were required to write detailed model descriptions based on the Pyomo scripts of the optimization models. If a model was found to be infeasible, the expert participants were also responsible for diagnosing the source of the infeasibility. Following these tasks, the experts were surveyed to provide time estimates for completing them. Given the differences in expertise among the participants and the varying complexity of the assigned models, we report the most frequently selected time range in the survey.

On the other hand, OptiChat automatically interprets optimization models in natural language without involving optimization experts, which significantly shortens the time. More importantly, both the experts and OptiChat are augmented with tools to isolate the irreducible infeasible subset (see the details in Section 3.1) for characterizing infeasible optimization models. By isolating the conflicting constraints, optimization experts can gain valuable insight from the IIS analysis. However, it is still time consuming for them to identify root causes and take corrective actions. In contrast, OptiChat automates the troubleshooting process along with a model description within one minute as shown in Table 1. Empirically, the quality of OptiChat’s description is observed to be comparable with that provided by experts, which can be attributed to the versatility of LLMs in different problem contexts. Details of the expert survey used to evaluate the model descriptions generated by the LLM are provided in Appendix B.

Table 1. Model Description Results

In practical applications, the model description in natural language is often already available before the model code is developed, so having the LLM generate such descriptions is not always necessary. Nevertheless, this study serves two important purposes. First, it demonstrates the flexibility of the LLM in explaining why a model is infeasible when an infeasible instance is encountered after model development. Second, by comparing the LLM-generated descriptions with expert answers, we can validate that the outputs produced by the illustrator agent are sufficiently accurate and reliable to support subsequent query responses.

4.2. Query Response Assessment

After reading the autonomous model description, practitioners gain an understanding of the problem context and are prepared to interact with OptiChat by asking queries. Solution-specific queries are addressed in two steps. First, OptiChat either invokes predefined functions or generates code to draw accurate conclusions from the optimization models. Second, OptiChat explains these conclusions in natural language to guarantee their interpretability to practitioners.

4.2.1. Query Data Set Development.

We curate a comprehensive test data set of 172 question-answer pairs to quantitatively evaluate the accuracy of OptiChat in response to the user’s query, with each pair developed in the context of a particular optimization model. The queries and answers were drafted by the 29 experts recruited, and each was verified by multiple authors of the paper. We classify these questions into five categories: (1) diagnosing, (2) retrieval, (3) sensitivity, (4) what-if, and (5) why-not queries as exemplified in Figure 2.

The correctness rate of query responses is evaluated using two distinct approaches. For why-not queries that rely on code generation, the data set provides natural language explanations explicitly containing the optimal objective values of counterfactual models. An LLM is used to compare these ground-truth objective values with those produced by OptiChat. If the new optimal objective value computed by OptiChat matches the value specified in the ground-truth answer, the LLM marks the response as correct. In rare cases, however, the LLM may generate incorrect counterfactual constraints while still producing a matching objective value. To ensure reliability, we manually review instances marked correct by the LLM to validate both the generated constraints and the resulting answers. In contrast, diagnosing, retrieval, sensitivity, and what-if queries are addressed through predefined functions. These function calls return results that are directly informative to the query. Across all data set instances, we observe that the explainer agent does not hallucinate when interpreting function outputs, provided that the correct function and arguments (e.g., component names and indices) are selected. To enable automatic evaluation, ground-truth outputs are structured in JavaScript Object Notation (JSON) format. If both the function name and arguments chosen by OptiChat match the ground truth, the corresponding answers are deemed correct.

4.2.2. Quantitative Assessment of the Query Responses.

Table 2 reports results for four different LLMs: GPT-4o-mini, GPT-4o, GPT-4.1, and o3. The first three are standard pretrained LLMs. o3 is OpenAI’s latest reasoning model, which features chain-of-thought reasoning at the expense of longer “thinking” time. Among them, o3 achieves the highest average accuracy across all different types of queries. Notably, it achieves a significantly improved performance on the why-not queries—78.4% compared with 62.2% for GPT-4.1 and 54.1% for GPT-4o—highlighting the value of explicit reasoning capabilities when code generation is required to generate the counterfactual constraints. The trade-off is that o3 achieves higher accuracy but with greater latency, as “why-not” queries can take up to 0.9 minutes per response. In contrast, the model with the smallest size, GPT-4o-mini, underperforms across all categories, suggesting limitations in both scale and reasoning depth. Overall, OptiChat delivers strong accuracy (above 80% in most query types) and fast responses (under one minute), confirming its practicality for interactive use.

Table 2. The Accuracy and Average Response Time of the Five Types of Queries Using GPT-4o-mini, GPT-4o, GPT-4.1, and o3

The high accuracy of OptiChat is attributed to two key factors. First, we develop a multiagent framework to guide LLMs in generating the code for counterfactual constraints. The prompt is specifically tailored for why-not queries rather than general queries, guiding the LLM to generate a minimal amount of code under a narrowed task scope. Second, we incorporate various predefined functions to prevent LLMs from developing explanatory techniques from scratch for other queries. This approach is more robust than code generation as demonstrated by the higher correctness rates in the first four queries in Table 2. More importantly, code generation is currently unable to develop more advanced explanatory techniques necessary for diagnosing queries and sensitivity queries. It is observed that code generation only produces heuristic techniques to tackle these queries, which results in suboptimal conclusions. Therefore, predefined functions are indispensable at the current stage. These factors contributing to the high accuracy will be discussed in more detail in the ablation studies (Section 4.2.4).

4.2.3. Failure Analysis.

Despite achieving high accuracy, the LLM may occasionally fail to match a query to the appropriate function or generate code that deviates from the user’s intended purpose. To better understand these model failures, we categorize errors into three types: syntax errors, classification errors, and logic errors.

Syntax errors include code execution failures and invalid function calls. These issues often arise from the ambiguity of natural language, which may mislead the LLM into generating a function argument or code that is not executable. Practitioners may describe the same model component using different terminology, whereas only one executable representation is valid in the model. This challenge is further amplified when components are indexed across multiple dimensions. Users may specify only a subset of dimensions, list them in an incorrect order, or omit them entirely. For example, pc[“max", :] denotes the maximum production capacities for all facilities, yet practitioners might refer to it verbally as max output limits, frequently omitting the qualifier for all facilities. The LLMs might hallucinate by providing indices that do not exist.

Classification errors occur when the explanation strategy selected by the LLM does not align with the human-annotated query type, such as misclassifying a why-not query as a what-if query.

Logic errors arise in two scenarios. First, during code generation, the LLM may fail to represent a counterfactual scenario correctly using constraints. Second, the component names or indices generated by the LLM may exist in the model as valid function arguments (i.e., not a syntax error) but fail to accurately capture the user’s intent.

With this categorization in mind, the breakdown of errors using GPT-4.1 and o3 is shown in Table 3. The proportions of syntax and classification errors in o3 (10.0% and 25.0%, respectively) are notably lower compared with GPT-4.1, which exhibits 46.8% syntax errors and 31.3% classification errors. This indicates that o3 adheres more reliably to the syntax guidance and few-shot query demonstrations provided in the prompts. As a result, the proportion of logic errors becomes dominant in o3 (65.0%) compared with GPT-4.1 (21.9%).

Table 3. Proportion of Error Types

4.2.4. Ablation Studies.

To evaluate the impact of the proposed multiagent framework, we perform ablation studies by removing the predefined functions, the syntax reminders, and the illustrator. The results are shown in Table 4 for the top two best-performing models: GPT-4.1 and o3.

Table 4. Ablation Study Results Including Experiments Without Predefined Functions, Syntax Reminders, and Illustrator

When the predefined functions are removed in Table 4, OptiChat must rely entirely on code generation (i.e., the programmer agent) to answer all queries. We observe a substantial drop in accuracy for both models, especially on the diagnosing and sensitivity queries. This indicates that the LLMs struggle to write code for retrieving the IIS or the dual variables of a constraint, tasks that demand deeper optimization expertise than the retrieval and what-if queries. Although both models suffer accuracy losses, o3 outperforms GPT-4.1, consistent with the claim that o3 is a superior reasoning model that excels at code generation. We also note a slight increase in response time without the predefined functions, reflecting the additional time that the LLMs spend generating code versus leveraging existing tools.

The syntax reminder provides guidance for retrieving the correct arguments for predefined functions. When the reminder is disabled, as shown in Table 4, we observe a slight accuracy drop in the o3 model and a significant drop in GPT-4.1. This further confirms the stronger reasoning capabilities of o3, whereas the weaker model tends to hallucinate without the syntax reminder. The difference in response time is marginal with or without the syntax reminder.

The illustrator agent extracts model components and their descriptions into a lookup table, providing essential context for answering various queries. This context is crucial for OptiChat’s performance. As shown in Table 4, both models experience a notable drop in accuracy without the illustrator. Moreover, the response time increases significantly because the models must reprocess the original code to answer follow-up queries rather than leveraging the pre-extracted context.

4.2.5. Qualitative Assessment of Query Responses.

We showcase some insightful answers produced by OptiChat. The models used for the demonstration are built for supply chain management, which decides the optimal production, storage, and transportation of goods across production facilities, distribution centers, and markets. The maximum capacities of the normal production and the overtime production often fluctuate because of factors such as raw material availability and labor engagement. Consider a case where a manufacturer is informed about an increase in raw materials and plans to expand maximum production capacities at certain facilities. As suggested in Figure 4, the optimal profit is highly sensitive to the maximum overtime production capacity at the third facility, even though overtime production is commonly perceived as costly. Consequently, during the strategy development phase, it is crucial to prioritize the third facility over others and produce these additional items at the overtime stage.

The second model takes into account recovery centers that manage remanufacturing and recovery activities. This model suggests building two such recovery centers, despite the substantial investment cost typically associated with each. Suppose a situation where business managers express concerns regarding the financial burden and are inclined to construct fewer recovery centers. To reassure them, OptiChat justifies this decision by demonstrating that no feasible solution exists if forcing the number of recovery centers to be no greater than one in Figure 5. Therefore, business managers can be convinced to deprecate outdated practices and appreciate the construction of recovery centers.

5. Conclusions and Future Work

In this paper, we propose OptiChat to address the common challenge that end users of optimization models are often not optimization experts. By leveraging the recent advancements in LLMs, OptiChat bypasses the need for inefficient back-and-forth coordination with optimization experts. Through straightforward natural language conversations with models, the practitioners can benefit from autonomous and instant responses. The integration of optimization tools with LLMs is demonstrated to be essential for providing a richer and more reliable context for analysis rather than relying entirely on natural language processing to generate plausible explanations.

Although OptiChat, at its current status, does not achieve perfect accuracy in executing tools or generating code, it is anticipated that this issue can be further mitigated by foreseeable advancements in LLM technology and targeted supervised fine-tuning specific to optimization tasks. One important step in this direction is the development of a larger and more diverse testing data set to better evaluate and improve OptiChat’s performance. Data augmentation techniques, such as prompting the LLM to rephrase the same questions, can be incorporated. Furthermore, users may exhibit varied preferences for answers and explanations depending on their background and application context, especially in multiturn conversations. To accommodate this, recent progress in reinforcement learning from human feedback offers a promising pathway. By training a reward model based on user preferences, we can better align OptiChat’s responses with the diverse expectations of its users. A more robust method can also be developed to evaluate users’ satisfaction with OptiChat in multiturn conversations. Last but not least, more explainable optimization techniques, such as inherently interpretable policies in the form of decision trees (Bertsimas and Stellato 2021, Goerigk and Hartisch 2023), can be added to the OptiChat. Thanks to OptiChat’s modular design, incorporating new explanation strategies or tool sets can be achieved with minimal overhead. We believe that OptiChat along with the new data set and the insights into query classification could serve as a catalyst in this understudied area and draw more attention from the optimization and machine learning community.