Comparing Exploration–Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-Armed Bandit Experiments

Description of Software and Data

The code and data in the zip file referenced above are a snapshot of the software and data that were used in the research reported in the paper "Comparing E&E Strategies of LLMs and Humans" by Ziyuan Zhang, Darcy Wang, Ningyuan Chen, Rodrigo Mansur, and Vahid Sarhangian. This repository is also available via Github.

The goal of this repository is to replicate the numerical experiments in the paper.

Computer and Software Environment

The following describes the computer hardware conditions and software environment on which the authors produce the results reported in the paper.

The results reported in the paper were produced using PyTorch 2.0.1 (CUDA 11.8 build) in Python 3.10.6 on Windows 11 (64-bit), running on an NVIDIA GeForce RTX 3060 Laptop GPU (6 GB, compute capability 8.6; CUDA 11.8, cuDNN 8.7) and an AMD Ryzen 9 5900HS with Radeon Graphics (3.30 GHz).

Dependencies

The code in this repository requires the following dependencies. The dependency version number corresponds to the version of the package with which the code was tested.

• Python 3.12.4
• arviz>=0.21.0
• cmdstanpy>=1.2.5
• numpy>=2.2.4
• pandas>=2.2.3
• anthropic>=0.49.0
• google.genai>=1.27.0
• openai>=1.66.2
• requests>=2.32.3
• matplotlib>=3.9.0
• scipy>=1.13.1

Installation

Download Python.

Run following command: > pip install -r requirements.txt

Use cmdstanpy.install_cmdstan() to install stan, it can be run as follows in terminal:

 >python
>>> import cmdstanpy
>>> cmdstanpy.install_cmdstan()

Reproducibility Workflow

To reproduce the results in Collection of the LLM-behavior dataset for the non-stationary 4-armed bandit (Section 3.2)

• Data File: dataCollection/reward_template.npy
• Code File: dataCollection/4_arm_data_collection.ipynb
• Output: Model-specific 4-arm .npy data files

To reproduce the results in Collection of the LLM-behavior dataset for the stationary 2-armed bandit (Section 3.2)

• Data File: Prompts and experiment settings embedded in the notebook
• Code File: dataCollection/2_arm_data_collection.ipynb
• Output: Model-specific merged .csv data files

To reproduce the results in Collection of UCB, epsilon-greedy, and Thompson-sampling decisions for the 2-armed bandit

• Code File: dataCollection/2_arm_baseline_model.ipynb
• Output: data/2armsdata/300experements/baseline/*.csv
• Run Time at the Above-Specified Computer Conditions: 6 min

To reproduce the results in Table 2 parameter-recovery study

• Code File: Table2_parameter_recovery_extracted.ipynb
• Run Time at the Above-Specified Computer Conditions: ~15 h

To reproduce the results in Table EC.1(b); Table EC.4; Tables EC.7-EC.10; Figures 4-5

• Data File: Explicit data/4armsdata/*.npy paths listed in the command file
• Code File: scripts/stan/cell_table_ec1b_ec4_ec7_ec10.sh; fitModel/runStan4arm.py; stanfiles/4arms/beta.stan; stanfiles/4arms/beta_phi.stan; stanfiles/4arms/beta_phi_rho.stan; analysis/plotData.ipynb
• Output: fittedModel/4_arms/**/*.csv and sibling *_loo.txt files
• Run Time at the Above-Specified Computer Conditions: ~70h

To reproduce the results in Table EC.1(a); Table EC.3; Tables EC.11-EC.16; Figures 2-3

• Data File: Explicit published non-human data/2armsdata/*.csv paths listed in the command file; human data excluded
• Code File: scripts/stan/cell_table_ec1a_ec3_ec11_ec16.sh; fitModel/runStan2arm.py; stanfiles/2arms/beta.stan; stanfiles/2arms/beta_phi.stan; stanfiles/2arms/beta_phi_rho.stan; stanfiles/2arms/probit.stan; analysis/plotData.ipynb
• Output: fittedModel/2_arm/**/*.csv and sibling *_loo.txt files
• Run Time at the Above-Specified Computer Conditions: ~7h

To reproduce the results in Table 3; Figure 7; Table EC.2

• Data File: Published non-human data/2armsdata/*.csv files; human data excluded
• Code File: analysis/regret_and_expl_2arm.py; analysis/plotData.ipynb
• Output: analysis/2_arm_true_regret.csv and the corresponding report outputs
• Run Time at the Above-Specified Computer Conditions: 1 s

To reproduce the results in Figures 6 and 8; Table EC.4

• Data File: data/4armsdata/*.npy
• Code File: analysis/regret_and_expl_4arm.py
• Output: analysis/4_arm_true_regret.csv; analysis/4_arm_expl.csv
• Run Time at the Above-Specified Computer Conditions: 1 s

To reproduce the results in Figures 9-10

• Data File: data/2armsdata/300experements/defult_temp/*.csv
• Code File: scripts/stan/cell_figures_9_10.sh; fitModel/runStanSwapSubjectBlock50.py; fitModel/runStanSwapSubjectBlock100.py; fitModel/runStanSwapSubjectBlock150.py; fitModel/runStanSwapSubjectBlock.py; stanfiles/2arms/beta_phi_rho.stan; analysis/plot_2_arm_different_rounds.py
• Output: fittedModel/2_arm/differentRounds/{50,100,150,300}/**/*.csv; sibling *_loo.txt files; figures/2_arm_300_no_thinking.png; figures/2_arm_300_thinking.png
• Run Time at the Above-Specified Computer Conditions: 2h30m

To reproduce the results in Figure 11

• Data File: data/2armsdata/300experements/defult_temp/*.csv
• Code File: analysis/plot_regret.py
• Output: figures/fig_cumregret_ucb.png
• Run Time at the Above-Specified Computer Conditions: 4 min

To reproduce the results in Figure 12

• Data File:data/2armsdata/300experements/defult_temp/*.csv
• Code File: analysis/qcare_estimation.py
• Output: figures/fig1_alpha_estimates.png and associated QCARE result files
• Run Time at the Above-Specified Computer Conditions: 31 s

To reproduce the results in Figure 13

• Data File:data/2armsdata/300experements/{defult_temp,high_temp,low_temp}/*.csv
• Code File: roc_pixelevel.py2% col: eval_contam_levels=False; 5/8/10% cols: eval_contam_levels=True (both eval_rdae_variants=False)
• Output: Default/high/low-temperature fitted summary CSVs and sibling *_loo.txt files; final figure manually assembled
• Run Time at the Above-Specified Computer Conditions: 2h30m

To reproduce the results in Figure 14

• Data File:data/4armsdata default/high/low-temperature .npy files listed in the command file
• Code File: scripts/stan/cell_figure_14.sh; fitModel/runStan4arm.py; stanfiles/4arms/beta_phi_rho.stan; manual figure extraction
• Output: Default/high/low-temperature fitted summary CSVs and sibling *_loo.txt files; final figure manually assembled
• Run Time at the Above-Specified Computer Conditions: 2h30m

To reproduce the results in Figures EC.3-EC.4; Tables EC.5-EC.6

• Data File:data/2armsdata/300experements/baseline/*.csv
• Code File: test_RobMemAE.py (whether_plot=True; set model_running + category per contam level)
• Output: fittedModel/baseline300results/beta_phi_rho/model/*.csv; sibling *_loo.txt files; figures/EG_beta.png; figures/UCB_beta.png
• Run Time at the Above-Specified Computer Conditions: 4h

Note:

Data for figure 2 – 5 were manually extracted from the table in EC.7 and EC.11. In LLM data collection please only run one block under the choose agent section, and make suer to fill out the api keys with your own api keyboth fitModel/runStan4arm.py and fitModel/runStan2arm.py takes in three arguments, in following order:

• *modelPath*: the path to the stan model (should end with `.stan`)
• *dataPath* = the data used to fit the model (end with `.npy` for 4-arms task and `.csv` for 2 -arms task)
• *savePath* = the path where the parameters of the fitted model is stored in, please ensure the path end with `.csv`, it will also generate a folder for the fitted stan model with the same name, and a txt file that records the cross validation data.

Cite

To cite the contents of this repository, please cite both the paper and this repository using their respective DOIs.

Article: https://doi.org/10.1287/ijds.2025.0113
Software and Data Repository: https://doi.org/10.1287/ijds.2025.0113

License

Copyright (c) (2026 Zhang, Wang, Chen, Mansur, Sarhangian)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.