# Similarity Experiment Replication Package

This directory contains the analysis code and data needed to replicate the
similarity experiment results. All scripts read from a single pre-processed dataset and
produce figures, tables, and statistical output.

## Setup

All scripts use `here()` for path management, anchored to the project root
(where the `.Rproj` file lives). R package dependencies are managed via
[rv](https://github.com/A2-ai/rv). Before running any scripts, run from the
project root:

```bash
./setup.sh
```

Scripts also depend on the `shared/` folder at the project root level:
- `shared/utils.R` — data loading and processing functions (used by 8 analysis scripts)
- `shared/hash_userId.R` — standalone hashing utility for anonymizing input datasets

## Paper Figures

| Figure | Script | Output | Verified |
|--------|--------|--------|----------|
| Fig 1B: Iteration plot (cosine sim by DALL-E version + ATE) | `similarity_exp_replication/code/analysis/figure1_iterPlots_PROD.R` | `similarity_exp_replication/outputs/figures/fig1_iterPlot_stratified.pdf` | ✅ |
| Fig 1C: Prompt length by iteration (colored by similarity) | `similarity_exp_replication/code/analysis/prompting.R` | `similarity_exp_replication/outputs/figures/prompt_length_similarity_by_model.pdf` | ✅ |
| Fig 2A: Replay decomposition (Model Effect + Prompting Effect) | `similarity_exp_replication/code/analysis/model_replay.R` | `similarity_exp_replication/outputs/figures/replay_decomposition_base_model_2.pdf` | ✅ |

## Paper Tables

| Table | Script | Output | Verified |
|-------|--------|--------|----------|
| Table 1: Distributional Effects of Model Upgrades in the Image Replication Task | `similarity_exp_replication/code/analysis/model_replay.R` | `similarity_exp_replication/outputs/tables/qte_table.tex` | ✅ |
| Table A.1: Covariate Balance Across All Three Arms (Image Replication) | `similarity_exp_replication/code/analysis/prod_balance_tbl_main_exp.R` | `similarity_exp_replication/outputs/tables/balance_table_main_exp_*.csv` | ✅ |

## Pre-Registered Hypotheses

| Hypothesis | Script | Description | Output |
|------------|--------|-------------|--------|
| H1 | `similarity_exp_replication/code/analysis/prereg_hypothesis1.R` | ANOVA: demographics/skills vs performance | console |
| H2 | `similarity_exp_replication/code/analysis/prereg_hypothesis2.R` | NLP features as performance predictors (matching) | console |
| H3 | `similarity_exp_replication/code/analysis/prereg_hypothesis3.R` | Combined demographic + NLP regression | console |
| H4 | `similarity_exp_replication/code/analysis/prereg_hypothesis4.R` | Iteration-level post-stratification | console |
| H5 | `similarity_exp_replication/code/analysis/prereg_hypothesis5_PROD.R` | Formatted regression tables with contrasts | console |
| H6 | `similarity_exp_replication/code/analysis/prereg_hypothesis6_PROD.R` | ANOVA + quantile treatment effects | `similarity_exp_replication/outputs/h5-h6/`, `similarity_exp_replication/outputs/tables/pnas-h6/` |
| H7 | `similarity_exp_replication/code/analysis/prereg_hypothesis7_PROD.R` | Blocked/stratified two-sample tests | console |
| H8 | `similarity_exp_replication/code/analysis/prereg_hypothesis8_PROD.R` | Baseline performance visualization | console |

## Input Data

`similarity_exp_replication/inputs/main_df_promptLvl_2026-03-17_PROD_HASHED.csv` is the analysis-ready
dataset (18,930 rows, 333 columns). Each row is one prompt-iteration for one
participant. User IDs are hashed for anonymization. The dataset includes:

- Cosine similarity and DreamSim performance scores (original + replicated)
- NLP features: prompt length, TSR, cosine similarity between successive prompts
- Topical transition metrics (cumulative topic counts, transition flags)
- Qualtrics survey responses (demographics, skills, AI experience)
- Treatment assignment (DALL-E version: 2, 3-verbatim, 3-revised)

## Running

From the project root in R or RStudio:

```r
source(here::here("similarity_exp_replication", "code", "analysis", "prereg_hypothesis1.R"))
```

Or from the command line:

```bash
Rscript similarity_exp_replication/code/analysis/prereg_hypothesis1.R
```

Output directories are created automatically where needed.

## Directory Structure

```
similarity_exp_replication/
├── inputs/
│   └── main_df_promptLvl_2026-03-17_PROD_HASHED.csv
├── code/
│   └── analysis/         # 12 scripts (see tables above)
├── outputs/              # generated by running scripts
│   ├── figures/
│   ├── h5-h6/
│   └── tables/           # LaTeX table outputs (e.g., qte_table.tex)
│       └── pnas-h6/
└── logs/
```
