# CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models

## Overview
This repository contains the official codebase for the paper **CrowdLLM: Building LLM-Based Digital
Populations Augmented with Generative Models**.

CrowdLLM is a novel framework that integrates pretrained Large Language Models (LLMs) with generative models. Unlike existing methods that rely solely on LLMs, CrowdLLM captures both the shared consensus and the intrinsic variance of real human crowds, creating highly accurate and diverse digital populations.

Our framework is evaluated across three applications:

* **Crowdsourcing**: Simulating annotator behaviors on subjective tasks (Offensiveness & QA Difficulty).

* **Recommendation**: Predicting user preferences on e-commerce platforms (Amazon Beauty & Music).

* **Voting**: Replicating complex human voting mechanisms (Zurich PB Voting).

## Data Preparation

Our experiments evaluate CrowdLLM across three applications: Crowdsourcing, Recommendation, and Voting. Below is the detailed breakdown of the raw data sources and the processed files for each task.

### Crowdsourcing
This task includes the Offensiveness and Question Answering (QA) Difficulty datasets.
    
1. Raw Data
   
    We utilize the POPQUORN dataset introduced in the following paper:

    >    Pei J, Jurgens D (2023) When do annotator demographics matter? measuring the influence of annotator demographics with the POPQUORN dataset. Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII), 252–265.

- Download: The raw data can be downloaded from the official repository: https://github.com/Jiaxin-Pei/potato-prolific-dataset.

- Storage: Please place the downloaded raw files into the `data/raw/offensiveness/` and `data/raw/qadifficulty/` directory.

2. Processed Data

- Training data: We process the raw files using our provided script `scripts/prepare_crowdsourcing_data.py`. The resulting files are stored in `data/processed/offensiveness/` and `data/processed/qadifficulty/`:

    - `offensiveness_data.pkl` / `qadifficulty_data.pkl`: The cleaned dataset containing texts, user IDs, question IDs, and extracted text embeddings.

    - `offensiveness_demo_name.npy` / `qadifficulty_demo_name.npy`: A serialized array of the demographic feature names (e.g., race, gender).

- Pre-computed LLM Baselines: To reduce API costs and facilitate reproducibility, we provide pre-evaluated LLM baseline results (Gemma, Qwen, LLaMA) in the processed folder:

    - `test_answer_byllm.pkl`: Static, single-round LLM predictions used for calculating final Distribution Metrics (Cosine Similarity and Wasserstein Distance).

    - `test_result_LLM_mae.pkl`: Pre-calculated MAE metrics scaled from 1 to 50 simulated workers, used specifically for plotting the Arena Evaluation curves.

    - `llm_test_questions.pkl`: Multi-iteration LLM responses used as the baseline for the Cost Efficiency experiment (Problems Resolved vs. API Calls).

### Recommendation
This task includes the Amazon Beauty and Amazon Music subsets.

1. Raw Data

    Extracted from the Amazon Reviews 2023 dataset.

    >   Hou Y, et al. (2024). Amazon Reviews 2023 Dataset.
McAuley J, et al. (2015). Image-based recommendations on styles and substitutes. SIGIR.

- Download: The full dataset is available at https://amazon-reviews-2023.github.io/.

- Storage Note: Due to the extremely large size of the raw dataset, we do not download the full files into the folder.

2. Processed Data

    We have already sampled and processed the necessary subsets, which are directly available in `data/processed/beauty/` and `data/processed/music/`:

   - Beauty Dataset: A subset of the "All Beauty" category where each product has 20 to 30 distinct users (448 products, 11,154 reviews, 10,957 unique users).

   - Music Dataset: A filtered subset of the "Musical Instruments" category strictly containing products reviewed by exactly 20 distinct users (629 products, 12,580 reviews, 12,396 unique users).

### Voting

This task utilizes the Zurich PB Voting dataset.

1. Raw Data

    Conducted in March 2023, this dataset contains 180 participants evaluating 24 urban projects under various voting rules. We specifically adopt the B (S5): Select 5 projects rule.

    >   Yang JC, Dalisan D, Korecki M, Hausladen CI, Helbing D (2024a). LLM voting: Human choices and AI collective decision-making. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 1696–1708.

- Download: The raw JSON files can be downloaded from https://github.com/joshuay1/zurich-pb-voting.

- Storage: Please place the downloaded raw files into the `data/raw/voting/` directory.

2. Processed Data

    The raw JSON files are cleaned and transformed into tabular formats suitable for our voting evaluator. These are stored in `data/processed/voting/`:

    - `voting_data.csv`: The primary dataset containing the parsed voting records (0 or 1) and participant-project mappings.

    - `voting_demo_name.npy`: An array of the 12-dimensional predefined participant profiles (e.g., transportation preferences, environmental beliefs) used to construct the context vector.

    - `exp1_voting_detail.csv`: A detailed, item-level prediction log comparing human ground truth against various model ablations (e.g., Fixed Belief, Fixed Blender). This file is specifically used by the evaluator to generate the overlaid frequency ablation charts.

## CrowdLLM Framework Implementation

The `src/` directory contains the core implementation of the CrowdLLM architecture.

### Core Modules

These scripts define the fundamental neural network components that power the CrowdLLM architecture:

- `ProfileGenerator.py`: Manages the extraction and dynamic injection of user demographic and predefined profile features (context vectors) into the generative process.

- `BeliefGenerator.py`: Implements the VAE-based generator that learns and samples the latent distribution of user beliefs.

- `Blender.py`: Implements the blender module that seamlessly integrates the baseline reference answer, the task embedding, and the sampled latent beliefs to output the final simulated preference.

### Digital Worker

These modules assemble the atomic generative components and external APIs into agents:

- `CrowdLLMWorker.py`: Integrates the BeliefGenerator, Blender, and ProfileGenerator into a cohesive worker class capable of autonomously simulating human responses.

- `LLMWorker.py`: A dedicated wrapper class for handling queries to external large language model baselines (e.g., LLaMA, Qwen, Gemma), facilitating structured prompting strategies like zero-shot and multi-persona inference.

### Unified Training & Task-Specific Evaluation

- `trainer.py`: Contains the unified training pipeline.

- `evaluators.py`: Contains the task-specific evaluation. Because the three applications require different testing mechanisms, the evaluation pipelines are decoupled into specialized evaluator classes (e.g., CrowdLLMEvaluator, RecomEvaluator, VotingEvaluator, and SimulationEvaluator).

### Auxiliary Utilities

`data_utils.py`, `Functions.py`, `CustomData.py`: Provide essential helper functions for dataset parsing, customized PyTorch DataLoader construction, metric calculations, and visualization plotting.

## How to Run

We provide four primary entry points (`main_*.py`) at the root directory. Each script is tailored to execute the complete pipeline (training, evaluation, and plotting) for a specific task.

### `main_crowdsourcing.py`

**Domain:** Crowdsourcing (Offensiveness Rating & QA Difficulty)

**What it does:** Trains CrowdLLM across different training data fractions (1%, 10%, 50%, 100%) and evaluates its performance against varied numbers of simulated workers using two distinct metrics.

**Key Findings Reproduced:**

**1. Arena Evaluation (Metric: Mean Absolute Error)**
* **Extreme Data Efficiency:** Evaluated by MAE against varied numbers of simulated workers, CrowdLLM demonstrates remarkable cost-saving potential. As shown in the figures below, it achieves performance comparable to the 100% full-data setting using only 1% of the human training data.
<p align="center">
<img src="assets/Offensive_MAE.png" width="48%" title="Offensiveness Dataset MAE" />
<img src="assets/Difficulty_MAE.png" width="48%" title="QA Difficulty Dataset MAE" />
</p>

**2. Cost Efficiency Evaluation (Metric: Proportion of Problems Resolved)**
* **The Value of Diversity:** Evaluated by the proportion of problems resolved against varied numbers of simulated workers, the results prove that generating virtual workers with diverse beliefs steadily improves task resolution rates. In contrast, scaling up workers with fixed, identical profiles fails to yield similar improvements.
<p align="center">
<img src="assets/Offensive_cost2.png" width="48%" title="Offensiveness Dataset Cost Efficiency" />
<img src="assets/Difficulty_cost2.png" width="48%" title="QA Difficulty Dataset Cost Efficiency" />
</p>

### `main_recommendation.py`

**Domain:** Product Ratings (Amazon Beauty & Music Instruments)

**What it does:** Trains the model using historical user reviews as textual profiles and evaluates the simulated preference distributions against real human crowds using both point-wise Accuracy Metrics (MAE, RMSE) and Distribution Metrics (CS, WD).

**Key Findings Reproduced:**

* **Balancing Accuracy and Diversity:** While LLMs (even with multi-persona prompting) exhibit limited diversity, CrowdLLM achieves the optimal balance. It successfully captures both diversity and accuracy, resulting in rating distributions that closely align with those of real human participants.


### `main_voting.py`

**Domain:** Voting

**What it does:** Integrates 12-dimensional predefined human profiles to train the model, strictly replicates the "Select 5 Projects" voting rule to generate final project-level vote counts, and evaluates these simulated outcomes against real human crowds using both point-wise Accuracy Metrics (MAE, RMSE) and Distribution Metrics (CS, WD).

**Key Findings Reproduced:**

**1. Voting Preference Distribution**
* **Capturing the "Long-Tail" of Human Preferences:** Traditional LLM-based synthetic crowds suffer from severe vote concentration (`#009E73` green bars), leaving less popular alternatives with zero votes. The full CrowdLLM framework successfully captures the sparse votes for less popular projects by allowing voters' intrinsic beliefs to be randomly generated (`#E15759` red bars), matching real human collective voting patterns (`#808080` grey bars).
<p align="center">
<img src="results/voting/bar_all_overlaid_freq.png" width="80%" title="Ablation Overlaid Frequency Chart" />
</p>


### `main_simulation.py`

**Domain:** Theoretical Performance Study (based on Offensiveness)

**What it does:** Executes Monte Carlo simulations by testing various combinations of data scale (number of workers and questions per worker) under dynamic Gaussian noise to generate a comprehensive 3x3 Simulation Matrix Chart.

**Key Findings Reproduced:**

**1. Simulation Robustness**
* **The "Sweet Spot" of Diversity:** The simulation demonstrates that a moderate level of belief diversity provides the best balance between cost and accuracy. If diversity is too low, individual errors are amplified rather than corrected when adding more workers; if diversity is too high, the model requires a massive amount of data to reach a stable conclusion. Moderate diversity avoids both extremes, delivering strong performance with significantly fewer data annotations.

<p align="center">
<img src="results/simulation/simulation_3x3_matrix.png" width="80%" title="Simulation Study" />
</p>