[![INFORMS Journal on Computing Logo](https://INFORMSJoC.github.io/logos/INFORMS_Journal_on_Computing_Header.jpg)](https://pubsonline.informs.org/journal/ijoc)

# Tackling Arbitrarily Heterogeneous Data in Asynchronous SGD without Worker Scheduling

This repository is distributed in association with the [INFORMS Journal on Computing](https://pubsonline.informs.org/journal/ijoc) under the [MIT License](LICENSE).

The software and data in this repository are associated with the paper:

**Xiaolu Wang, Yuchang Sun, Hoi-To Wai, and Jun Zhang**  
[*Tackling Arbitrarily Heterogeneous Data in Asynchronous SGD without Worker Scheduling*](https://doi.org/10.1287/ijoc.2025.1443)

## Cite

Please cite both the paper and this code repository.

**Paper DOI**  
https://doi.org/10.1287/ijoc.2025.1443

**Code repository DOI**  
https://doi.org/10.1287/ijoc.2025.1443.cd

```bibtex
@misc{wang2026dudecode,
  author    = {Xiaolu Wang and Yuchang Sun and Hoi-To Wai and Jun Zhang},
  title     = {Tackling Arbitrarily Heterogeneous Data in Asynchronous SGD without Worker Scheduling},
  year      = {2026},
  publisher = {INFORMS Journal on Computing},
  doi       = {10.1287/ijoc.2025.1443.cd},
  url       = {https://github.com/INFORMSJoC/2025.1443},
  note      = {Available at https://github.com/INFORMSJoC/2025.1443}
}
```

## Description

This repository contains the implementation of **Dual-Delayed SGD (DuDe-SGD)** together with the baseline methods used in the paper, including Delayed SGD, FedBuff, Uniform ASGD, Shuffled ASGD, MIFA, and synchronous baselines.

All source code in this repository was written by the authors for this project, unless otherwise noted.

The repository supports two main workflows:

- **Fashion-MNIST DuDe-SGD sensitivity experiments**, where DuDe-SGD is run under different values of the data-heterogeneity parameter `alpha`.
- **CIFAR-10 comparison experiments**, where DuDe-SGD is compared with multiple baselines under different values of `alpha`, system heterogeneity, and random seeds.

In addition, the codebase also supports **MNIST** as a standard benchmark for quick testing. MNIST is not part of the main experimental workflows reported in the paper.

## Repository Structure

```text
.
├── AUTHORS
├── LICENSE
├── README.md
├── requirements.txt
├── results/
│   ├── figures/
│   └── raw/
├── scripts/
│   ├── run.py
│   ├── run_DuDe.py
│   ├── draw_csv.py
│   ├── draw_csv_DuDe.py
│   ├── merge_logs_for_comparison.py
│   ├── exp_cifar_10workers.sh
│   └── exp_cifar_30workers.sh
└── src/
    ├── modules/
    ├── models/
    └── other_utils/
```

The `results/raw/` directory contains both per-run raw CSV logs (for example, `log_...-test-acc.csv` and `log_...-train-loss.csv`) and the merged seed-level CIFAR comparison CSV files used by `scripts/draw_csv.py` (for example, `cifar-acc-10-3407.csv`).

## Requirements

Install the Python dependencies with:

```bash
pip install -r requirements.txt
```

Recommended environment: Python 3.7 or 3.8 with the package versions listed in `requirements.txt`.

## Data Sources

The experiments use standard benchmark datasets retrieved through standard `torchvision` dataset loaders at runtime; the datasets themselves are not redistributed in this repository.

Relevant references are:

- **MNIST**: Yann LeCun, Corinna Cortes, and Christopher J. C. Burges. *The MNIST Database of Handwritten Digits*. 1998. Available at: http://yann.lecun.com/exdb/mnist/
- **Fashion-MNIST**: Han Xiao, Kashif Rasul, and Roland Vollgraf. *Fashion-MNIST: A Novel Image Dataset for Benchmarking Machine Learning Algorithms*. arXiv preprint arXiv:1708.07747, 2017.
- **CIFAR-10**: Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. *The CIFAR-10 Dataset*. Canadian Institute for Advanced Research / University of Toronto. Available at: https://www.cs.toronto.edu/~kriz/cifar.html

The experiments reported in this repository were first run around mid-2024.

## Main Workflows

### 1. Fashion-MNIST DuDe-SGD sensitivity experiments

Run the DuDe-SGD experiments under different values of `alpha`:

```bash
python scripts/run_DuDe.py
```

This script runs DuDe-SGD on Fashion-MNIST for the preset `alpha` values and writes raw log CSV files into `results/raw/`.

To visualize the resulting curves, use `draw_csv_DuDe.py`.

#### Plot DuDe-SGD test accuracy

```bash
python scripts/draw_csv_DuDe.py --lr 0.01 --comput-dist "normal(1,5)" --metric test-acc
```

#### Plot DuDe-SGD training loss

```bash
python scripts/draw_csv_DuDe.py --lr 0.01 --comput-dist "normal(1,5)" --metric train-loss
```

`draw_csv_DuDe.py` reads DuDe-SGD raw CSV files in `results/raw/` whose filenames match the selected learning rate and computation-speed distribution.

### 2. CIFAR-10 comparison experiments

Run the 10-worker CIFAR-10 comparison experiments:

```bash
bash scripts/exp_cifar_10workers.sh
```

Run the 30-worker CIFAR-10 comparison experiments:

```bash
bash scripts/exp_cifar_30workers.sh
```

These scripts run the raw experiments, write raw CSV logs into `results/raw/`, and then generate merged seed-level comparison CSV files in the same `results/raw/` directory.

To display the comparison figures from the merged CSV files stored in `results/raw/`, run:

```bash
python scripts/draw_csv.py --results-dir ./results/raw --n-workers 10 --metric both
```

or

```bash
python scripts/draw_csv.py --results-dir ./results/raw --n-workers 30 --metric both
```

## Notes on Reproducibility

- Randomness is controlled through `--init_seed`.
- The code retrieves MNIST, Fashion-MNIST, and CIFAR-10 automatically when needed through dataset loaders.
- The `--datadir` argument supports paths such as `~/data/`; the code expands `~` to the user home directory automatically.
- Some experiment scripts assume access to CUDA devices. If needed, replace `cuda:0`, `cuda:1`, etc., with devices available on your machine.
- The provided CIFAR-10 shell scripts launch multiple jobs in parallel. On machines with limited GPU memory, remove `&` from the scripts to run them sequentially.
- If `--save_net` is enabled, model checkpoints are written under the directory specified by `--modeldir`. The necessary checkpoint subdirectories are created automatically by the code when needed.
