Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection

Description of Software and Data

The code and data in the zip file referenced above are a snapshot of the software and data that were used in the research reported in the paper "Active Learning with Imperfect Labels: Optimal Labeler Assignment and Sample Selection" by Pouya Ahadi, Blair Winograd, Camille Zaug, Karunesh Arora, Lijun Wang, and Kamran Paynabar. All code except code specific to the Ford Motor Company case study (Section 4.4) is publicly available via GitHub.

The goal of this repository is to replicate the numerical experiments in the paper. As detailed in the Note section below, the case study in Section 4.4 of the paper uses proprietary data provided by Ford Motor Company under a confidentiality agreement; that dataset and the corresponding code are not included in this repository.

Random Seeds and Reproducibility

The train/test split (random_state=42) and the classifier initializations (random_state=0) use a fixed random state, held constant across all replications and methods, so these two elements are exactly reproducible. The remaining stochastic elements of each replication, generating the labeler accuracy pool, sampling the initial labeled set, selecting query points under random sampling, assigning the specific incorrect class when a label is corrupted, resolving ties in majority voting, and the k-means++ seeding used by BADGE+RLA, draw from the numpy and Python standard-library random number generators without a fixed seed, and are therefore not deterministically reproducible across separate executions of the code. This is reported here for transparency; it does not affect the validity of the paired significance tests reported in the paper, since those rely on the seven core methods being computed within the same per-replication execution, not on exact cross-execution reproducibility.

Computer and Software Environment

The following describes the computer hardware conditions and software environment on which the authors produce the results reported in the paper.

• Hardware: Apple Silicon (arm64), 10 CPU cores, 16 GB RAM..
• Operating System: macOS (Darwin 25.6.0).
• Python Version: 3.9.6.
• Parallelization: replications are parallelized across CPU cores using joblib; the number of cores used can be configured in each script.

Dependencies

• numpy 2.0.2
• scipy 1.13.1
• pandas 2.3.3
• scikit-learn 1.6.1
• joblib 1.5.3
• openpyxl 3.1.5

These are also listed in requirements.txt in the repository and can be installed with pip install -r requirements.txt.

Installation

1. Clone the repository: git clone https://github.com/AhadiPouya/OLASOptimal-Labeler-Assignment-and-Sampling-for-Active-Learning.git.
2. Install dependencies: pip install -r requirements.txt.
3. Download the public datasets (Statlog Heart, Ionosphere, Connectionist Bench Sonar, Spambase, and Covertype from the UCI Machine Learning Repository; the Music Genre Classification dataset from Rodrigues, Pereira, and Ribeiro 2013) into a local Data/folder.
4. Each script expects a project folder containing Data/ and Results/ subfolders. By default this is a folder named OLAS_Results in the current working directory; this can be changed by setting the OLAS_BASE_DIR environment variable.

File structure: the repository is a flat directory of six Python scripts at the top level (source code), plus requirements.txt and LICENSE. There are no data files included in the repository; scripts read from, and write results to, the Data/ and Results/ subfolders described above, which the user creates locally.

Reproducibility Workflow

To reproduce the results in Table 1 (dataset summary), Table 2 (experimental settings), for the four smaller UCI datasets

• Data File: Statlog Heart, Ionosphere, Sonar, Spambase (UCI)
• Code File: olas_uci_exp eriments.py
• Output: Table 1 and Table 2 contents (printed/logged)
• Run Time at the Above-Specified Computer Conditions: On the order of minutes

To reproduce the results in Table 3 (Noise Model 1 results) and Table 4 (Noise Model 2 results), for the four smaller UCI datasets

• Data File: Statlog Heart, Ionosphere, Sonar, Spambase (UCI)
• Code File: olas_uci_experiments.py
• Output: F1 scores per method/replication, saved as .xlsx files; aggregated into Table 3/4
• Run Time at the Above-Specified Computer Conditions: Roughly 5 hours (serial CPU time; shorter with parallelization)

To reproduce the results in Table 3 and Table 4, Covertype row

• Data File: dCovertype (UCI)
• Code File: olas_covertype_optimized.py
• Output: F1 scores per method/replication for Covertype, saved as .xlsx
• Run Time at the Above-Specified Computer Conditions: Roughly 2.5 hours (serial CPU time)

To reproduce the results in Table 3, 4, and 5, BADGE+RLA rows (all five datasets)

• Data File: Statlog Heart, Ionosphere, Sonar, Spambase, Covertype (UCI)
• Code File: olas_badge_runner.py
• Output: F1 scores and per-cycle timings for BADGE+RLA, saved as .xlsx (dominated by the Covertype run)
• Run Time at the Above-Specified Computer Conditions: Roughly 6.5 hours (serial CPU time; dominated by Covertype)

To reproduce the results in Table 5 (average computation time per AL cycle)

• Data Files: Same as above
• Code File: olas_uci_experiments.py, olas_covertype_optimized.py, olas_badge_runner.py
• Output: Per-cycle wallclock times, logged during the runs above
• Run Time at the Above-Specified Computer Conditions: Captured during the runs above; no separate run needed

To reproduce the results in Section 4.1 text, alpha sensitivity results (Statlog Heart and Spambase, alpha in {0.15, 0.20, 0.25})

• Data Files: Statlog Heart, Spambase (UCI)
• Code File: olas_alpha_sensitivity.py
• Output: F1 scores by alpha value, saved as .xlsx
• Run Time at the Above-Specified Computer Conditions: On the order of tens of minutes

To reproduce the results in Section 3.4 validation figure (fitted noise function and beta threshold), Section 4.3 text and Figures (median F1 plot, box plot)

• Data Files: Music Genre Classification dataset (Rodrigues et al. 2013)
• Code File: music_pipeline.py
• Output: music_pipeline_fit.jpg, and the fitted noise-model parameters/estimates reported in Section 4.3
• Run Time at the Above-Specified Computer Conditions: On the order of tens of minutes

To reproduce the results in Appendix C (paired/unpaired significance tests)

• Data Files: Result .xlsx files produced by the scripts above
• Code File: olas_significance_tests.py
• Output: paired_significance_results.xlsx, printed summary table
• Run Time at the Above-Specified Computer Conditions: Under one minute

Figure 4 (illustrative plot of Noise Model 2) is a small supplementary plot generated directly from the closed-form Noise Model 2 formula given in Section 4.1 of the paper; it is not part of the experimental pipeline above and is not included as a separate script in this repository, but can be regenerated directly from that formula.

Note:

The case study in Section 4.4 of the paper (Ford Motor Company warranty claim data, Figures 8 and 9) uses proprietary data provided under a confidentiality agreement with Ford Motor Company. Neither the data nor the code specific to that case study is included in this repository, consistent with IJDS’s data and code disclosure policy for proprietary data. The methodology applied in that case study is the same OLAS framework implemented in the public scripts above (olas_uci_experiments.py in particular implements the same core optimization and assignment logic used for the Ford case study), applied to a differently structured, non-public dataset.

Run times listed above are approximate, derived from the average per-active-learning-cycle times reported in Table 5 of the paper and the number of replications and cycles used for each experiment; they assume serial execution and were not independently re-measured end to end. Actual wall-clock time will be shorter when replications are run in parallel (see the joblib-based parallelization in each script) and may vary with different hardware.

Cite

To cite the contents of this repository, please cite both the paper and this repository using their respective DOIs.

Article: https://doi.org/10.1287/ijds.2025.0105
Software and Data Repository: https://doi.org/10.1287/ijds.2025.0105.cd

License

Copyright (c) (Ahadi, Winograd, Zaug, Arora, Wang, Paynabar)

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.