Using Large Language Models to Create Analytics Solutions

The Value and Risks of LLMs

Large language models (LLMs) such as ChatGPT have made novel advanced artificial intelligence (AI) capabilities widely available through natural language. (There is even initial research toward using LLMs to make advanced OR/MS capabilities available to nonexperts; see [1, 2].) These advanced LLM capabilities already benefit individuals in many jobs and roles, including knowledge workers. A study by the Harvard Business School [3] showed that for workers carrying out complex business consulting tasks, consultants who had access to LLMs completed 12% more tasks on average, completed their tasks 25% more quickly and produced 40% higher quality results. However, these benefits are not without risk: The study also revealed that when consultants used LLMs for a task for which the LLM was not suitable, even high-performing consultants exhibited diminished performance compared with those who did not use unsuitable LLMs. This finding suggests a potential tendency to excessively rely on the LLM’s output. Furthermore, numerous LLMs introduce additional challenges, such as generating erroneous outputs with high confidence. Another issue associated with LLM usage is the potential ambiguity surrounding the intellectual property rights of its output, because LLMs are often trained on internet data, some of which may be subject to copyright. Additionally, privacy concerns arise as the inputs provided to LLMs can frequently be used for further training and could be exposed to additional LLM users.

Can LLMs Provide Value in the Creation of Analytics Solutions?

Developing an analytics solution to address real-world problems is a multistep process that demands deep expertise in analytics and a range of soft skills. This process requires an analytics practitioner to engage with the business problem owner, comprehend the problem, construct an appropriate model, enable the business owner to understand the solution and be convinced of its value, and then deploy the solution into a complex environment involving multiple stakeholders. Moreover, the solution may need to evolve as the problem or environment changes. As a result, creating such an analytics solution is a lengthy process, and many analytics application projects fail. In such an environment, any tool that can significantly reduce the solution creation time while improving the quality of the provided solution is highly desirable. Given the potential benefits and risks of LLMs, two important questions arise:

1. What value, if any, can LLMs contribute to the analytics solution creation process?
2. Assuming they can indeed provide value, how should LLMs be appropriately used to ensure the solution quality is not adversely affected and to avoid any additional issues inherent in their usage?

The Potential Value of LLMs for Creating Analytics Solutions

To begin answering these questions, our workgroup [4], led by the INFORMS Practice Section, conducted the following experiment: We interacted with ChatGPT-4 in natural language to carry out a mock walkthrough of the creation of an analytics solution for Mobian Global [5] – a Netherlands-based company seeking to optimize the placement of rental hubs for bikes (bikes used by commuters and tourists from outside the city to travel in the city). The solution was previously successfully created without the use of an LLM by a group led by Prof. Dick den Hertog, who also participated in this work. This allowed Prof. den Hertog to evaluate both the potential value of ChatGPT and recognize any mistakes or issues in the implementation.

As a structured process for the solution creation, we used the INFORMS Certified Analytics Professional Job Task Analysis (JTA), a multistep iterative framework that provides analytics practitioners with an overview of the analytics process to ensure data-driven outcomes. (The appropriate usage of the JTA was informed by Dr. Arnie Greenland, CAP, another member of our group, who was a significant contributor to the JTA definition). The JTA comprises multiple domains: Business Problem Framing, Analytic Problem Framing, Data, Methodology Framing, Model Building, Deployment and Model Lifecycle Management (see Figure 1). Each domain consists of multiple tasks (for more details, see [5]).

We found that ChatGPT could indeed be used to provide input and insights for most of the tasks in the JTA across all domains. Moreover, ChatGPT’s output could potentially provide value both in improving the solution quality and in shortening the time required to improve the solution throughout the JTA process.

Regarding improving solution quality, we found that ChatGPT provided a comprehensive list of domain- and problem-specific considerations across the entire JTA lifecycle. This included, among others:

• A comprehensive list of relevant problem-specific stakeholders (see Figure 2), required as part of the Business Problem Framing JTA domain.
• A comprehensive set of possible analytical techniques relevant to the problem as part of the Analytic Problem Framing domain.
• A comprehensive set of relevant data types and sources as part of the Data domain.
• Comprehensive guidelines and validation criteria for the solution as a part of the Deployment domain.

These capabilities can improve solution quality and success rates by addressing key considerations. They can also help analytics teams quickly become knowledgeable about new domains and techniques, increasing the chances that the best analytical approach is used for each project, even if the analytics practitioner is initially unfamiliar with a given technique.

For shortening solution creation time, ChatGPT was able to very quickly generate relevant problem-specific content, including creating initial optimization models in the Model Building domain, generating initial data-cleansing code in the Data domain, generating initial web UI code and generating documents relevant to the process, such as concise problem statements as part of the Business Problem Framing domain and user manual outlines for the Deployment domain.

In addition, we found that ChatGPT can potentially be extremely helpful in quickly and accurately deriving the core analytical problem underlying the business problem (part of the Business Problem Framing domain), thereby potentially significantly accelerating the practitioner’s understanding of the problem and its correct formulation using analytics.

The Role of the Analytics Practitioner

Our experiment demonstrated the potential for LLMs to serve as a significant accelerator and quality enhancer throughout the creation of an analytics solution. However, it also emphasizes the importance and required deep expertise of the analytics practitioner in this process. We found that practitioners need to maintain full control and ensure validation when using LLMs. For instance, when initially queried about stakeholders, ChatGPT only listed external stakeholders and had to be explicitly queried to provide a list of internal ones. Furthermore, ChatGPT did not manage to precisely define the users – it output that commuters were an important user group and defined a commuter as any person traveling from home to work, but for this specific problem, commuters relevant to Mobian’s bike rentals are only commuters living outside the city. Another example is that the initial optimization model generated by ChatGPT included an implicit assumption that all users entering a city through a specific junction and traveling to a specific point of interest should use the same bicycle hub, whereas this assumption was not required.

Therefore, when using LLMs, the practitioner’s role includes:

• Ensuring a thorough understanding of the problem.
• Maintaining ultimate responsibility for creating models suitable for the business problem.
• Determining, for each comprehensive list of items generated by LLMs, which items are indeed relevant to the business problem and which should be disregarded (e.g., data that is not needed should not be collected just because the LLM suggests it).
• Ensuring that no relevant aspects have been neglected throughout the JTA, because although LLMs are proficient at providing a comprehensive set of items and considerations, exhaustiveness is not guaranteed.
• Validating the correctness and appropriateness of any suggestion made by LLMs to help mitigate issues such as generation of erroneous output.

Guidelines for the Appropriate Usage of LLMs

Our experiments indicate that LLMs can be a powerful tool in the analytics practitioners’ toolbox. However, practitioners must treat it like any other tool and ensure that they:

• Understand how to use it and learn skills, such as how to craft appropriate natural language inputs, to obtain the best results (a skill commonly known as prompt engineering).
• Recognize that, owing to the nature of this tool, validating the correctness and appropriateness of the output is of utmost
• Become aware of any potential intellectual property (IP) and privacy issues associated with the LLM used and addresses them The practitioner should especially be mindful of what data and information is uploaded to the tool and consider obtaining approval from the party for whom the analytics solution is being created.
• Become aware of the high financial and environmental cost associated with the user of LLMs, due to the extensive computational resources they require.

Summary

Our experiments show that an LLM can be used extensively throughout the creation of an analytics application to augment the analytics practitioner and potentially significantly improve the quality and speed of solution delivery. However, ultimately, to ensure appropriate usage, the practitioner must retain overall responsibility and ownership for the analytical solution, treat this as just another tool in the toolbox, and use the LLM, like any other tool, in a responsible and ethical manner.

References

1. AhmadiTeshnizi, W. Gao and M. Udell, 2023, “OptiMUS: Optimization modeling using MIP solvers and large language models,” https://doi.org/10.48550/arXiv.2310.06116.
2. Wasserkrug, L. Boussioux, D. den Hertog, F. Mirzazadeh, I. Birbil, J. Kurtz and D. Maragno, 2024, “From large language models and optimization to decision optimization copilot: A research manifesto,” https://arxiv.org/html/2402.16269v1.
3. Dell’Acqua, E. McFowland, E. R. Mollick, H. Lifshitz-Assaf, K. Kellogg, S. Rajendran et al., 2023, “Navigating the jagged technological frontier: Field experimental evidence of the effects of ai on knowledge worker productivity and quality,” Harvard Business School Technology & Operations Mgt. Unit Working Paper (24-013).
4. The workgroup members include Leonard David Jean Boussioux; Aaron Burciaga, CAP; Dick den Hertog; Pooja Dewan; Fred Garrett; Arnie Greenland, CAP; Michael Haydock; Shailendra Jain; Rajeev Namboothiri; Patricia Neri; Pascal van Hentenryck; and Segev Wasserkrug.
5. Institute for Operations Research and the Management Sciences (INFORMS). INFORMS Job Task Analysis. https://www.certifiedanalytics.org/jta.
6. https://www.mobian.global/en