This folder contains all evaluation outputs generated by `train.py`
and `evaluate.py`. Each subfolder corresponds to one model variant.


## Running All Ablations

All outputs are saved automatically to `results/` by running the commands below.
Each subfolder contains `best_params.json`, `metrics_by_seed.csv`,
`metrics_summary.csv`, and `test_predictions_by_seed.csv`.

## Ablation 1: Shot Segmentation vs. Sentence Segmentation

The paper uses shot-based segmentation as the default, where transcripts are
segmented using screen-change boundaries and sequences of words are
assigned based on timestamps. Sentence segmentation (NLTK) is the
ablation baseline used for comparison.

Re-run the data pipeline with the appropriate `--segmentation` flag
(see main README Step 1), then run Steps 2-4 pointing to the
corresponding embeddings.

| Segmentation      | Pipeline flag              | csv_dir                   | Output prefix  |
|-------------------|----------------------------|---------------------------|----------------|
| Shot-based (ours) | `--segmentation shots`     | `data/Transcripts_Shots/` | none           |
| Sentence (ablation)| `--segmentation sentences` | `data/Transcripts/`       | `Baseline/` (auto-set) |

> **Note:** Zenodo embeddings use shot-based segmentation only.
> Sentence-based embeddings must be generated from scratch using Step 1.

```bash
python scripts/data_pipeline.py --stages all \
    --json_dir       data/sample/JSON \
    --csv_dir        data/sample/Transcripts \
    --video_dir      data/sample/Videos \
    --frames_dir     data/sample/Video_Frames \
    --manifest       data/sample/video_manifest_sentences.txt \
    --segmentation   sentences \
    --text_encoder   bert \
    --vision_encoder vit \
    --no-local_only
```

**Then sentence segmentation outputs automatically go to:**
```
Embeddings/Baseline/BERT_ViT/Final/softmax/Co_Attn/
data/Attention_Files/Baseline/BERT_ViT/
data/Attention_Scores/Baseline/BERT_ViT/attention_scores.csv
```

```bash

# Step 2: Grid search for sentence-level segmentation
python scripts/train.py \
    --variant         co_attn \
    --pt_dir          Embeddings/Baseline/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/Baseline/BERT_ViT/attention_scores.csv \
    --test_json       data/test_videoids.json \
    --out_dir         results/Baseline/BERT_ViT/softmax/Co_Attn \
    --epochs 50 --patience 5 --batch_size 32 --seed 42 --device cpu

# Step 3: Evaluate for sentence-level segmentation
python scripts/evaluate.py eval \
    --pt_dir          Embeddings/Baseline/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/Baseline/BERT_ViT/attention_scores.csv \
    --test_json       data/test_videoids.json \
    --model_dir       results/Baseline/BERT_ViT/softmax/Co_Attn \
    --seeds 42 --epochs 50 --patience 5 --device cpu

# Step 4: Compare shot-based (co_attn) with sentence-based (co_attn)
python scripts/evaluate.py compare \
    --preds1  results/BERT_ViT/softmax/Co_Attn/test_predictions_by_seed.csv \
    --preds2  results/Baseline/BERT_ViT/softmax/Co_Attn/test_predictions_by_seed.csv \
    --n_ids   100 --n_iters 100 \
    --out_csv results/BERT_ViT/softmax/compare_baseline_coattn.csv

```

---

## Ablation 2: Input and Attention Variants

No pipeline rerun needed — all modality folders are already generated
by Stage 7 of `data_pipeline.py`. Just point `--pt_dir` to the
relevant subfolder and set `--variant` accordingly.

### Base Inputs

| Model        | `--variant` | `--pt_dir`                                  |
|--------------|-------------|---------------------------------------------|
| Text-only    | `text`      | `Embeddings/BERT_ViT/Final/softmax/Text`    |
| Frames-only  | `frames`    | `Embeddings/BERT_ViT/Final/softmax/Frames`  |

### Attention Variants

| Model              | `--variant` | `--pt_dir`                                      |
|--------------------|-------------|-------------------------------------------------|
| Text→Frame         | `t2f`       | `Embeddings/BERT_ViT/Final/softmax/T2F`         |
| Frame→Text         | `f2t`       | `Embeddings/BERT_ViT/Final/softmax/F2T`         |
| Concat             | `concat`    | `Embeddings/BERT_ViT/Final/softmax/concat`      |
| Mean               | `mean`      | `Embeddings/BERT_ViT/Final/softmax/mean`        |
| Co-Attention (ours)| `co_attn`   | `Embeddings/BERT_ViT/Final/softmax/Co_Attn`     |

### Example: run grid search and evaluate for each variant

```bash
# Step 2: Grid search (repeat for each variant)
python scripts/train.py \
    --variant         frames \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/Frames \
    --annotations_csv data/complete_set.csv \
    --test_json       data/test_videoids.json \
    --out_dir         results/BERT_ViT/softmax/Frames \
    --epochs 50 --patience 5 --batch_size 32 --seed 42 --device cpu

# Step 3: Evaluate (repeat for each variant)
python scripts/evaluate.py eval \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/Frames \
    --annotations_csv data/complete_set.csv \
    --test_json       data/test_videoids.json \
    --model_dir       results/BERT_ViT/softmax/Frames \
    --seeds 42 --epochs 50 --patience 5 --device cpu
```
> **Note:** Co-Attention requires `--attention_csv` in both train.py and evaluate.py. All other variants do not. 
---

## Ablation 3: Encoder Pair and Fusion Variants

Re-run the data pipeline with a different encoder pair (see main README
Step 1), then run Steps 2–4 with the matching `--pt_dir` and `--out_dir`.

### Intermediate Fusion (Co_Attn vs mean late vs concat) across Encoder Pairs

Compares how text and visual features are fused at the **intermediate level**
before final pooling. All results use `softmax` final pooling.

| Encoder Pair | Intermediate Fusion | variant flag | pt_dir                                        | out_dir                                        |
|--------------|---------------------|--------------|-----------------------------------------------|------------------------------------------------|
| BERT + ViT   | Co-Attention        | `co_attn`    | `Embeddings/BERT_ViT/Final/softmax/Co_Attn`   | `results/BERT_ViT/softmax/Co_Attn`             |
| BERT + ViT   | Mean fusion         | `mean`       | `Embeddings/BERT_ViT/Final/softmax/mean`      | `results/BERT_ViT/softmax/mean`                |
| BERT + ViT   | Concat fusion       | `concat`     | `Embeddings/BERT_ViT/Final/softmax/concat`    | `results/BERT_ViT/softmax/concat`              |
| LONG + DINO  | Co-Attention        | `co_attn`    | `Embeddings/LONG_DINO/Final/softmax/Co_Attn`  | `results/LONG_DINO/softmax/Co_Attn`            |
| LONG + DINO  | Mean fusion         | `mean`       | `Embeddings/LONG_DINO/Final/softmax/mean`     | `results/LONG_DINO/softmax/mean`               |
| LONG + DINO  | Concat fusion       | `concat`     | `Embeddings/LONG_DINO/Final/softmax/concat`   | `results/LONG_DINO/softmax/concat`             |
| BERT + DINO  | Co-Attention        | `co_attn`    | `Embeddings/BERT_DINO/Final/softmax/Co_Attn`  | `results/BERT_DINO/softmax/Co_Attn`            |
| BERT + DINO  | Mean fusion         | `mean`       | `Embeddings/BERT_DINO/Final/softmax/mean`     | `results/BERT_DINO/softmax/mean`               |
| BERT + DINO  | Concat fusion       | `concat`     | `Embeddings/BERT_DINO/Final/softmax/concat`   | `results/BERT_DINO/softmax/concat`             |
| LONG + ViT   | Co-Attention        | `co_attn`    | `Embeddings/LONG_ViT/Final/softmax/Co_Attn`   | `results/LONG_ViT/softmax/Co_Attn`             |
| LONG + ViT   | Mean fusion         | `mean`       | `Embeddings/LONG_ViT/Final/softmax/mean`      | `results/LONG_ViT/softmax/mean`                |
| LONG + ViT   | Concat fusion       | `concat`     | `Embeddings/LONG_ViT/Final/softmax/concat`    | `results/LONG_ViT/softmax/concat`              |

---

### Final Pooling (softmax vs mean vs attn) across Encoder Pairs

Compares how intermediate CLS embeddings are pooled into the final video-level vector. All results use `Co_Attn` intermediate fusion.

| Encoder Pair | Final Pooling | variant flag | pt_dir                                        | out_dir                                        |
|--------------|---------------|--------------|-----------------------------------------------|------------------------------------------------|
| BERT + ViT   | softmax       | `co_attn`    | `Embeddings/BERT_ViT/Final/softmax/Co_Attn`   | `results/BERT_ViT/softmax/Co_Attn`             |
| BERT + ViT   | mean          | `co_attn`    | `Embeddings/BERT_ViT/Final/mean/Co_Attn`      | `results/BERT_ViT/mean/Co_Attn`                |
| BERT + ViT   | attn          | `co_attn`    | `Embeddings/BERT_ViT/Final/attn/Co_Attn`      | `results/BERT_ViT/attn/Co_Attn`                |
| LONG + DINO  | softmax       | `co_attn`    | `Embeddings/LONG_DINO/Final/softmax/Co_Attn`  | `results/LONG_DINO/softmax/Co_Attn`            |
| LONG + DINO  | mean          | `co_attn`    | `Embeddings/LONG_DINO/Final/mean/Co_Attn`     | `results/LONG_DINO/mean/Co_Attn`               |
| LONG + DINO  | attn          | `co_attn`    | `Embeddings/LONG_DINO/Final/attn/Co_Attn`     | `results/LONG_DINO/attn/Co_Attn`               |
| BERT + DINO  | softmax       | `co_attn`    | `Embeddings/BERT_DINO/Final/softmax/Co_Attn`  | `results/BERT_DINO/softmax/Co_Attn`            |
| BERT + DINO  | mean          | `co_attn`    | `Embeddings/BERT_DINO/Final/mean/Co_Attn`     | `results/BERT_DINO/mean/Co_Attn`               |
| BERT + DINO  | attn          | `co_attn`    | `Embeddings/BERT_DINO/Final/attn/Co_Attn`     | `results/BERT_DINO/attn/Co_Attn`               |
| LONG + ViT   | softmax       | `co_attn`    | `Embeddings/LONG_ViT/Final/softmax/Co_Attn`   | `results/LONG_ViT/softmax/Co_Attn`             |
| LONG + ViT   | mean          | `co_attn`    | `Embeddings/LONG_ViT/Final/mean/Co_Attn`      | `results/LONG_ViT/mean/Co_Attn`                |
| LONG + ViT   | attn          | `co_attn`    | `Embeddings/LONG_ViT/Final/attn/Co_Attn`      | `results/LONG_ViT/attn/Co_Attn`                |

---

### Example: run one encoder pair combination
```bash
# Step 1: Generate embeddings for LONG+DINO pair
python scripts/data_pipeline.py --stages all \
    --json_dir       data/sample/JSON \
    --csv_dir        data/sample/Transcripts_Shots \
    --video_dir      data/sample/Videos \
    --frames_dir     data/sample/Video_Frames \
    --manifest       data/sample/video_manifest.txt \
    --segmentation   shots \
    --text_encoder   longformer \
    --vision_encoder dinov2

# Step 2: Grid search
python scripts/train.py \
    --variant         co_attn \
    --pt_dir          Embeddings/LONG_DINO/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/LONG_DINO/attention_scores.csv \
    --test_json       data/test_videoids.json \
    --out_dir         results/LONG_DINO/softmax/Co_Attn \
    --epochs 50 --patience 5 --batch_size 32 --seed 42 --device cpu

# Step 3: Evaluate
python scripts/evaluate.py eval \
    --pt_dir          Embeddings/LONG_DINO/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/LONG_DINO/attention_scores.csv \
    --test_json       data/test_videoids.json \
    --model_dir       results/LONG_DINO/softmax/Co_Attn \
    --seeds 42 --epochs 50 --patience 5 --device cpu
```
> Repeat for each row in the tables above by changing
> `--text_encoder`, `--vision_encoder`, `--pt_dir`, and `--out_dir`.

---

## Ablation 4: Variation in Training Sample Size

Default training size is 1149 records (80/20 train/val split of 1436 non-test videos).
All variations use the same fixed test set and val set (287 records).
When increasing to 1249, 100 records are moved from val to train.
When reducing, records are randomly dropped from train and val stays at 287.

| Training Size | Direction  | `--n_train` | `--out_dir`                                    |
|---------------|------------|-------------|------------------------------------------------|
| 1249          | +100       | `1249`      | `results/BERT_ViT/softmax/Co_Attn_n1249`       |
| 1149          | default    | omit        | `results/BERT_ViT/softmax/Co_Attn`             |
| 1049          | -100       | `1049`      | `results/BERT_ViT/softmax/Co_Attn_n1049`       |
| 949           | -200       | `949`       | `results/BERT_ViT/softmax/Co_Attn_n949`        |
| 849           | -300       | `849`       | `results/BERT_ViT/softmax/Co_Attn_n849`        |
| 749           | -400       | `749`       | `results/BERT_ViT/softmax/Co_Attn_n749`        |
| 649           | -500       | `649`       | `results/BERT_ViT/softmax/Co_Attn_n649`        |

```bash
# Step 2: Grid search (repeat for each n_train)
python scripts/train.py \
    --variant         co_attn \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/BERT_ViT/attention_scores.csv \
    --test_json       data/test_videoids.json \
    --out_dir         results/BERT_ViT/softmax/Co_Attn_n1249 \
    --epochs 50 --patience 5 --batch_size 32 --seed 42 --n_train 1249 --device cpu

# Step 3: Evaluate (repeat for each n_train)
python scripts/evaluate.py eval \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/BERT_ViT/attention_scores.csv \
    --test_json       data/test_videoids.json \
    --model_dir       results/BERT_ViT/softmax/Co_Attn_n1249 \
    --seeds 42 --epochs 50 --patience 5 --n_train 1249 --device cpu

# Step 4: Compare each training size with default (1149)
python scripts/evaluate.py compare \
    --preds1  results/BERT_ViT/softmax/Co_Attn/test_predictions_by_seed.csv \
    --preds2  results/BERT_ViT/softmax/Co_Attn_n1249/test_predictions_by_seed.csv \
    --n_ids   100 --n_iters 100 \
    --out_csv results/BERT_ViT/softmax/compare_1149_vs_1249.csv
```

> Repeat Steps 2-4 for each row in the table above, changing `--n_train` and `--out_dir` accordingly.

## Ablation 5: COVID 100 Videos (Test)

```bash
# Step 1: Generate embeddings for BERT_ViT pair
python scripts/data_pipeline.py --stages all \
    --json_dir       data/sample/COVID/JSON \
    --csv_dir        data/sample/COVID/Transcripts_Shots \
    --video_dir      data/sample/COVID/Videos \
    --frames_dir     data/sample/COVID/Video_Frames \
    --manifest       data/sample/COVID/video_manifest.txt \
    --segmentation   shots \
    --text_encoder   bert \
    --vision_encoder vit \
    --prefix COVID
```

**Outputs will be:**
```
Embeddings/COVID/BERT_ViT/Final/softmax/Co_Attn/   
data/Attention_Scores/COVID/BERT_ViT/attention_scores.csv  
```
> **Step 2: Skip grid search** if you already ran it for Co_Attn
> in the main README. The existing `results/BERT_ViT/softmax/Co_Attn/best_params.json`
> is reused automatically by Step 5.

```bash
# Step 3: Create COVID test JSON
python -c "
import json, os
ids = [os.path.splitext(f)[0]
       for f in sorted(os.listdir('Embeddings/COVID/BERT_ViT/Final/softmax/Co_Attn'))
       if f.endswith('.pt')]
json.dump(ids, open('data/covid_test_videoids.json','w'), indent=2)
print(f'Saved {len(ids)} COVID video IDs')
"

# Step 4: merge annotations (Original + COVID)
python -c "
import pandas as pd
df = pd.concat([
    pd.read_csv('data/complete_set.csv'),
    pd.read_csv('data/annotations_COVID19.csv')
], ignore_index=True)
df.to_csv('data/complete_set_with_covid.csv', index=False)
"

# Step 5: Evaluate on COVID test set
python scripts/evaluate.py eval \
    --pt_dir             Embeddings/BERT_ViT/Final/softmax/Co_Attn \
    --pt_dir_extra       Embeddings/COVID/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv    data/complete_set_with_covid.csv \
    --attention_csv      data/Attention_Scores/BERT_ViT/attention_scores.csv \
    --attention_csv_extra data/Attention_Scores/COVID/BERT_ViT/attention_scores.csv \
    --test_json          data/covid_test_videoids.json \
    --model_dir          results/BERT_ViT/softmax/Co_Attn_COVID \
    --seeds              42 \
    --epochs 50 --patience 5 --device cpu
```
---
**What happens internally:**
```
pt_dir        → 1795 original videos
pt_dir_extra  → 100 COVID videos
test_json     → covid_test_videoids.json (100 COVID IDs)
─────────────────────────────────────────────────────
train/val     → original non-test videos 
test          → 100 COVID videos only
results       → saved to Co_Attn_COVID/ (completely separate)
```
---

## Ablation 6: Video Duration (Short vs. Long) 

```bash
# Step 1 — Run split_test_by_duration.py to segregate videos by duration
# 240 represents seconds (number of frames as frame are split per 1 second)
python scripts/split_test_by_duration.py \
    --test_json  data/test_videoids.json \
    --frames_dir data/Video_Frames \
    --threshold  240 \
    --out_dir    data/
```

**Outputs:**
```
data/test_videoids_short.json   # videos < 240 frames
data/test_videoids_long.json    # videos >= 240 frames
```

```bash
# Step 2: Short videos only
python scripts/evaluate.py eval \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/BERT_ViT/attention_scores.csv \
    --test_json       data/test_videoids_short.json \
    --model_dir       results/BERT_ViT/softmax/Co_Attn_short \
    --seeds           42 \
    --epochs 50 --patience 5 --device cpu

# Step 2: Long videos only
python scripts/evaluate.py eval \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/BERT_ViT/attention_scores.csv \
    --test_json       data/test_videoids_long.json \
    --model_dir       results/BERT_ViT/softmax/Co_Attn_long \
    --seeds           42 \
    --epochs 50 --patience 5 --device cpu

# Step 3: Compare short vs. long videos
python scripts/evaluate.py compare \
    --preds1  results/BERT_ViT/softmax/Co_Attn_short/test_predictions_by_seed.csv \
    --preds2  results/BERT_ViT/softmax/Co_Attn_long/test_predictions_by_seed.csv \
    --n_ids   100 --n_iters 100 \
    --out_csv results/BERT_ViT/softmax/compare_short_vs_long.csv
```
