AMLB 2023Q2 Results

This repository contains the visualization notebooks and data for the 2023 benchmark. This is work in progress.

Structure

This repository maintains the following structure, root refers to the project root directory. Files and directories marked with an asterisk (*) may also be generated from notebooks in root/notebooks.

root
 |-- data
 |    |-- amlb_2021Q3.csv   Cleaned results from the 2021Q3 benchmark 
 |    |-- amlb_2023Q2.csv   All results from the 2023Q2 benchmark
 |    `-- * amlb_all.csv    Generated file with cleaned results of 2023Q2 combined with results from 2021Q3
 |
 |-- * figures                directory for subdirectories with all generated figures
 |      |-- * CD              CD plots
 |      |-- * performance     scaled performance boxplots
 |      |-- * duration        training duration boxplots
 |      `-- * inference       scatter plots of performance v. inference time
 |
 `-- notebooks
      |-- raw_to_clean.ipynb   Generates root/data/amlb_all.csv
      |-- critical_difference.ipynb  Generates CD plots in root/figures
      |-- visualization.ipynb  Generates other in root/figures
      |-- data_processing.py   Data processing helpers used in visualization.ipynb
      `__ visualization.py     Visualization helpers used in visualization.ipynb

Installation

The following setup is tested on a M1 Mac with Python 3.9. Setup a virtual environment:

python -m venv venv
source venv/bin/activate

install dependencies:

python -m pip install --upgrade pip wheel
python -m pip install -r requirements.txt
python -m pip install "Orange3==3.30.0" --no-deps

note: newer versions of Orange3 do not have critical difference plots, but 3.30.0 doesn't seem to install well out of the box. Hence the workaround above. It works for Python 3.9 but fails for Python 3.11.

starting the jupyter server:

jupyter-lab