## Datasets

This project uses one synthetic dataset and two publicly available datasets. Due to the sheer size of the dataset, we cannot archive all the raw data here. They can be easily generated through the code we provided in the folder `src/gen_data.py` or downloaded from opensourced website.



### Publicly Available Datasets

#### CIFAR-10 Dataset
The CIFAR-10 dataset is partitioned to build each client's local datasets according to the Dirichlet allocation with parameter $\alpha$; the smaller $\alpha$, the more non-IID of the local data. We choose $\alpha =0.1,$ which creates highly non-IID local datasets due to label skewness. 

The raw dataset can be accessed through [`torchvision`](https://docs.pytorch.org/vision/main/generated/torchvision.datasets.CIFAR10.html).


#### Shakespeare Datasets
The Shakespeare dataset is built from *The Complete Works of William Shakespeare*, which contains 4,226,158 data instances. 

The raw dataset can be accessed through [LEAF](https://leaf.cmu.edu/), which is a federated learning benchmark and is under The BSD 2-Clause License. It partitioned the dataset into 660 groups. We sample around 18\% from the 660 groups to obtain the local datasets for $M = 125$ clients. LEAF is under the license of BSD 2-Clause License and allows unrestricted use, modification, and distribution of the software. The license file is included in the folder of `src/leaf/data/shakespeare/LICENSE.md`.

### Synthetic Datasets
We generate the local dataset $(x\_{ij},y\_{ij})\_{j=1}^{n\_i}$ for each client $i$ according to the model $y\_{ij}=\arg\max (\text{softmax}(W\_i x\_{ij} + b\_i),$ where $x\_{ij}\in\mathbb{R}^{60}),$ $W_i\in\mathbb{R}^{10\times 60},$ $b_i\in\mathbb{R}^{10}.$ To generate heterogeneous clients, each element of $W\_i$ and $b\_i$ is independently drawn from $\mathcal{N}(u\_i,1)$, where $u\_i\sim \mathcal{N}(0,1)$. Moreover, $x\_{ij} \sim \mathcal{N}(v\_i,\Sigma),$ where the covariance matrix is diagonal with $\sum\_{j,j}=j^{-1.2}.$
Each element of the mean vector $v\_i$ is independently drawn from $\mathcal{N}(B\_i,1),$ where $B_i\sim \mathcal{N}(0,1).$
In contrast to the experiments on CIFAR-10, here we consider the setup with quantity skewness, where the local data volume $n_i$ follows a power law.

One demonstration synthetic dataset, generated under a fixed random seed and including both train and test datasets, is put under the folder `data/Synthetic`.