[![INFORMS Journal on Computing Logo](https://INFORMSJoC.github.io/logos/INFORMS_Journal_on_Computing_Header.jpg)](https://pubsonline.informs.org/journal/ijoc)

# CAAC: Co-attentive Actionability Classification for Assessing Patient Education Videos

This archive is distributed in association with the [INFORMS Journal on Computing](https://pubsonline.informs.org/journal/ijoc) under the [MIT License](LICENSE).

The software and data in this repository are a snapshot of the software and data that were used in the research reported on in the paper:
[CAAC: Co-attentive Actionability Classification for Assessing Patient Education Videos](https://doi.org/10.1287/ijoc.2023.0493) by Krishna Pothugunta, Xiao Liu, Anjana Susarla and Rema Padman.

**This code is being developed on an on-going basis at
https://github.com/pothugun/CAAC-Actionability. Please go there if you would like to
get a more recent version or would like support.**

## Cite

To cite the contents of this repository, please cite both the paper and this repo, using their respective DOIs.

https://doi.org/10.1287/ijoc.2023.0493

https://doi.org/10.1287/ijoc.2023.0493.cd

Below is the BibTex for citing this snapshot of the repository.

```
@misc{kpothugunta2026caac,
  author = {K. Pothugunta and X. Liu and A. Susarla and R. Padman}, 
  publisher = {INFORMS Journal on Computing},
  title = {{CAAC: Co-attentive Actionability Classification for Assessing Patient Education Videos}},
  year = {2026}, 
  doi = {10.1287/ijoc.2023.0493.cd},
  url = {https://github.com/INFORMSJoC/2023.0493},
  note = {Available for download at https://github.com/INFORMSJoC/2023.0493},
}  
```

## Description

This repository contains the implementation of CAAC, a co-attention-based multimodal approach for assessing actionability in patient education videos using PEMAT guidelines.

## Data

| File | Description |
|------|-------------|
| `data/complete_set.csv` | Video metadata and actionability labels |
| `data/test_videoids.json` | Fixed test split — held out from all training |
| `data/Attention_Scores/BERT_ViT/attention_scores.csv` | Pre-computed entropy scores |
| `data/sample/` | A sample video to verify the pipeline end-to-end |

Pre-computed embeddings are available on Zenodo:
[https://doi.org/10.5281/zenodo.18854006](https://doi.org/10.5281/zenodo.18854006)

**Note:** The Zenodo repository currently contains the following pre-computed embeddings for the BERT+ViT encoder pair (softmax pooling, 1795 videos):

  - Co-Attention (Co_Attn) — primary method
  - Text-to-Frame (T2F) — attention variant
  - Frame-to-Text (F2T) — attention variant
  - Text-only — base input
  - Frame-only — base input

These cover the core ablation results reported in Table 7 of the paper. To generate all embeddings from scratch, follow Step 1 in **replication**.


## Dependencies

This project requires:

- Conda (Miniconda or Anaconda)
- Python 3.12
- GPU + CUDA 12.4 recommended for embedding and entropy scores extraction (Stages 1-8)
- CPU is sufficient for grid search and evaluation (Steps 2-4)

## Building

### 1. Create environment

```bash
conda create --name myenv python=3.12 -y
conda activate myenv
```

### 2. Install dependencies

```bash
python -m pip install --upgrade pip

# CPU only:
conda install -y pytorch torchvision torchaudio cpuonly -c pytorch

# OR GPU (do not run both):
conda install -y pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia

conda install -c conda-forge fsspec
pip install -U "sympy==1.13.1"
pip install -r requirements.txt
```

### 3. Verify environment

```bash
python -c "import torch, sklearn, transformers; \
    print('torch', torch.__version__); \
    print('sklearn', sklearn.__version__); \
    print('transformers', transformers.__version__)"
```

> Note: PyTorch 2.5.1 requires sympy==1.13.1 specifically.


## Repository Structure

```
CAAC/
├── scripts/
│   ├── data_pipeline.py             # Stages 1-8: video → embeddings → entropy scores
│   ├── train.py                     # Grid search → best_params.json
│   ├── evaluate.py                  # Test evaluation + paired t-test comparison
│   ├── split_test_by_duration.py    # Split test set by video length ( < / >= 240 seconds)
├── data/
│   ├── complete_set.csv             # Video annotations (video_id, Actionable label)
│   ├── annotations_COVID19.csv      # Annotations for 100 COVID-19 videos (video_id, Actionable label)
│   ├── test_videoids.json           # Fixed held-out test set (never used in grid search)
│   ├── sample/
│   │   ├── Videos/
│   │   │   └── sample_video_1.mp4
│   │   │
│   │   ├── JSON/
│   │   │    └── sample_video_1.json
│   │   │   
│   │   ├── Transcripts_Shots/    # Generated by Stage 1 (run data_pipeline.py)
│   │   ├── Video_Frames/         # Generated by Stage 2 (run data_pipeline.py)
│   │   └── video_manifest.txt    # Lists sample video path (run data_pipeline.py)
│   └── Attention_Scores/
│       └── BERT_ViT/
│           └── attention_scores.csv   # Pre-computed entropy scores (Stage 8)
├── results/
│   └── README.md             # Commands to reproduce all ablation results
├── requirements.txt
├── AUTHORS
└── LICENSE
```

## Replication

```
Step 1  data_pipeline.py    Raw videos → embeddings and entropy scores
Step 2  train.py            Grid search → best_params.json
Step 3  evaluate.py eval    best_params.json → test AUC, F1, Precision, Recall
Step 4  evaluate.py compare Paired t-test between two variants
```
If you are using the **Zenodo embeddings**, skip Step 1 and start from Step 2.

## Step 1: Data Pipeline (skip if using Zenodo embeddings)

The pipeline processes raw videos through 8 sequential stages:

```
Stage 1  Segment transcripts using screen-change boundaries
Stage 2  Extract frames (1 fps)
Stage 3  Download pre-trained models
Stage 4  Compute text + frame embeddings (BERT/LongFormer, ViT/DINOv2)
Stage 5  Compute co-attention (T2F, F2T, Co_Attn)
Stage 6  Compute simple fusion (mean, concat)
Stage 7  Pool embeddings per video (softmax, mean, attn)
Stage 8  Compute entropy scores from attention maps	 
```

A sample video is provided in `data/sample/` to verify the pipeline
end-to-end before running on the full dataset.

### Run the full pipeline

On first run, include `--no-local_only` to download pre-trained encoders:

```bash
python scripts/data_pipeline.py --stages all \
    --json_dir       data/sample/JSON \
    --csv_dir        data/sample/Transcripts_Shots \
    --video_dir      data/sample/Videos \
    --frames_dir     data/sample/Video_Frames \
    --manifest       data/sample/video_manifest.txt \
    --segmentation   shots \
    --text_encoder   bert \
    --vision_encoder vit \
    --no-local_only
```

After the first run, models are cached and drop `--no-local_only`:

```bash
python scripts/data_pipeline.py --stages all \
    --json_dir       data/sample/JSON \
    --csv_dir        data/sample/Transcripts_Shots \
    --video_dir      data/sample/Videos \
    --frames_dir     data/sample/Video_Frames \
    --manifest       data/sample/video_manifest.txt \
    --segmentation   shots \
    --text_encoder   bert \
    --vision_encoder vit
```

The `--segmentation` flag accepts `shots` or `sentences`. The `--text_encoder` and `--vision_encoder` flags accept `bert`, `longformer`, `vit`, `dinov2`. See `results/README.md` for the segmentation (shots vs. sentences) and all encoder pair combinations.

### Supported encoder pairs

| Text Encoder | Vision Encoder | Pair Name |
|--------------|----------------|-----------|
| BERT         | ViT            | BERT_ViT  |
| LongFormer   | DINOv2         | LONG_DINO |
| BERT         | DINOv2         | BERT_DINO |
| LongFormer   | ViT            | LONG_ViT  |

Embeddings are saved to:
```
Embeddings/<PairName>/Final/<fusion>/<modality>/<video_id>.pt
```
Attention scores saved to:
```
data/Attention_Scores/<PairName>/attention_scores.csv
```

## Step 2: Grid Search (Hyperparameter Tuning)

Train on the train/val split only. The test set is **never used here**.

### CAAC (180 combinations: lr x wd x alpha x beta)

```bash
python scripts/train.py \
    --variant         co_attn \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/BERT_ViT/attention_scores.csv \
    --test_json       data/test_videoids.json \
    --out_dir         results/BERT_ViT/softmax/Co_Attn \
    --epochs          50 \
    --patience        5 \
    --batch_size      32 \
    --seed            42 \
    --device          cpu
```

### Other variants (20 combinations: lr x wd only)

```bash
python scripts/train.py \
    --variant         t2f \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/T2F \
    --annotations_csv data/complete_set.csv \
    --test_json       data/test_videoids.json \
    --out_dir         results/BERT_ViT/softmax/T2F \
    --epochs          50 \
    --patience        5 \
    --batch_size      32 \
    --seed            42 \
    --device          cpu
```

Available variants: `co_attn`, `t2f`, `f2t`, `text`, `frames`, `mean`, `concat`

**Outputs** saved to `--out_dir`:
```
grid_search_results.csv   all combinations and their val_auc / val_f1
best_params.json          best hyperparameters selected by val_auc
```

## Step 3: Evaluate on Test Set

Reads `best_params.json` automatically. Trains from random initialization
for each seed and evaluates on the fixed held-out test set.

### CAAC: Co-Attention (primary method)

```bash
python scripts/evaluate.py eval \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/Co_Attn \
    --annotations_csv data/complete_set.csv \
    --attention_csv   data/Attention_Scores/BERT_ViT/attention_scores.csv \
    --test_json       data/test_videoids.json \
    --model_dir       results/BERT_ViT/softmax/Co_Attn \
    --seeds           42 \
    --epochs          50 \
    --patience        5 \
    --device          cpu
```

### Example ablation variant: T2F

```bash
python scripts/evaluate.py eval \
    --pt_dir          Embeddings/BERT_ViT/Final/softmax/T2F \
    --annotations_csv data/complete_set.csv \
    --test_json       data/test_videoids.json \
    --model_dir       results/BERT_ViT/softmax/T2F \
    --seeds           42 \
    --epochs          50 \
    --patience        5 \
    --device          cpu
```


**Outputs** saved to `--model_dir`:
```
metrics_by_seed.csv              per-seed AUC, F1, Precision, Recall
metrics_summary.csv              mean +/- std across seeds
test_predictions_by_seed.csv     per-video predictions (used for comparison)
```

See `results/README.md` for commands to reproduce all ablation variants.

## Step 4: Compare Methods (Paired t-test)

Runs a paired t-test between two variants using their saved predictions.

```bash
python scripts/evaluate.py compare \
    --preds1  results/BERT_ViT/softmax/Co_Attn/test_predictions_by_seed.csv \
    --preds2  results/BERT_ViT/softmax/T2F/test_predictions_by_seed.csv \
    --n_ids   100 \
    --n_iters 100 \
    --out_csv results/BERT_ViT/softmax/compare_CoAttn_vs_T2F.csv
```

Significance levels: `*` p<0.10  `**` p<0.05  `***` p<0.01

## Ongoing Development

This code is being developed on an on-going basis at the author's
[Github site](https://github.com/pothugun/CAAC-Actionability).

## Support

For support in using this data and code, please get in touch with the author (pothugun@msu.edu).