CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models

Description of Software and Data

The code and data in the zip file referenced above are a snapshot of the software and data that were used in the research reported in the paper "CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models" by Ryan Feng Lin, Keyu Tian, Hanming Zheng, Congjing Zhang, Li Zeng, and Shuai Huang. This repository is also available via Github.

The goal of this repository is to replicate the numerical experiments in the paper.

Computer and Software Environment

The system is configured with an Intel Core i9 processor and 64GB of RAM, utilizing an NVIDIA GeForce RTX 3090 GPU for hardware acceleration.

Dependencies

The code in this repository requires the following dependencies. The dependency version number corresponds to the version of the package with which the code was tested.

The code requires the following primary Python libraries. These versions correspond to the environment where the code was successfully tested:

• crowd_kit==1.4.1
• matplotlib==3.10.8
• numpy==2.4.4
• openai==2.30.0
• pandas==3.0.2
• PyYAML==6.0.3
• scikit_learn==1.8.0
• scipy==1.17.1
• seaborn==0.13.2
• torch==2.8.0
• tqdm==4.67.1

Installation

Full data can be downloaded from Github.

Step-by-step Instructions:

1. Clone the repository: git clone https://github.com/KeyuTian/CrowdLLM.git
2. Environment Setup: We recommend using Conda to manage the complex dependencies:
• conda create -n crowdllm python=3.9
• conda activate crowdllm
• pip install -r requirements.txt
3. Data Preparation:
• Basic datasets are in data/processed/.
• Large-scale datasets (Beauty/Music) must be reconstructed from the split .zip files in their respective folders using 7-Zip or WinRAR.
• File Structure:
  

  
  CrowdLLM/
  ├── data/
  │   ├── processed/          # Cleaned datasets for all tasks
  │   │   ├── beauty/         # Amazon Beauty (large files, split as .zip)
  │   │   ├── music/          # Amazon Music (large files, split as .zip)
  │   │   ├── offensiveness/  # Offensiveness task data
  │   │   ├── qadifficulty/   # QA Difficulty task data
  │   │   └── voting/         # Voting data
  │   └── raw/                # Original source files (if applicable)
  ├── results/                # Outputs
  ├── src/                    # Core implementation logic and modules
  ├── config/                 # Hyperparameter and environment configurations
  ├── .gitignore              # Git exclusion rules
  ├── main_crdsrc.py         # Entry point for Crowdsourcing task
  ├── main_recom.py           # Entry point for Recommendation task
  ├── main_simulation.py      # Entry point for Simulation experiments
  ├── main_voting.py          # Entry point for Voting task
  ├── README.md               # Project documentation and instructions
  └── requirements.txt        # Python dependency list
  

Reproducibility Workflow

To reproduce the results in Table 1, Figure 4(a), Figure 5(a)

• Data File: offensiveness_data.pkl, offensiveness_demo_name.npy
• Code File: main_crdsrc.py
• Output: Results in Table 1 and Figure 4(a), Figure 5(a)
• Run Time at the Above-Specified Computer Conditions: 5 minutes for training and testing, 2 hours for LLM inference

To reproduce the results in Table 2, Figure 4(b), Figure 5(b)

• Data File: qadifficulty_data.pkl, qadifficulty_demo_name.npy
• Code File: main_crdsrc.py
• Output: Results in Table 2 and Figure 4(b), Figure 5(b)
• Run Time at the Above-Specified Computer Conditions: 5 minutes for training and testing, 2 hours for LLM inference

To reproduce the results in Table 3, Figure 6

• Data File: amazon_beuty.csv
• Code File: main_recom.py
• Output: Results in Table 3 and Figure 6
• Run Time at the Above-Specified Computer Conditions: 2 minutes for training and testing, 4 hours for LLM inference

To reproduce the results in Table 4, Figure 7

• Data File: amazon_music.csv
• Code File: main_recom.py
• Output: Results in Table 4 and Figure 7
• Run Time at the Above-Specified Computer Conditions: 2 minutes for training and testing, 4 hours for LLM inference

To reproduce the results in Table 5, Figure 8

• Data File: voting_data.pkl, voting_demo_name.npy
• Code File: main_voting.py
• Output: Results in Table 5, Figure 8
• Run Time at the Above-Specified Computer Conditions: 2 minutes for training and testing, 15 minutes for LLM inference

To reproduce the results in Figure 9

• Data File: simulation.pkl
• Code File: main_crdsrc.py
• Output: Figure 9
• Run Time at the Above-Specified Computer Conditions: 4 hours

Note

Task Switching via Subject Parameter:

To reproduce the Offensiveness and QA Difficulty results, run main_crdsrc.py and change the subject parameter in the configuration to the task name. For the Recommendation task, run main_recom.py and update the subject parameter to switch between the Beauty and Music datasets.

Cite

To cite the contents of this repository, please cite both the paper and this repository using their respective DOIs.

Article: https://doi.org/10.1287/ijds.2025.0140
Software and Data Repository: https://doi.org/10.1287/ijds.2025.0140.cd

License

Copyright (c) (2026 Lin, Tian, Zheng, Zhang, Zeng, Huang)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.