Skip to content

Running Cross Validation on SLURM Cluster

Jump to bottom
Robert L. Logan IV edited this page Apr 7, 2021 · 3 revisions

Setup

To start, check out the cross validation branch:

git checkout cross-val

Next configure your python environment. General guidelines for setting it up are provided in the README, but in brief you should be able to run:

conda create -n autoprompt -y python=3.7 && conda activate autoprompt
conda install pytorch cudatoolkit=10.2 -c pytorch
pip install -e .

There might be some version mismatches with the adapter-transformers dependencies, but as far as I've been able to test they shouldn't affect anything.

Cross-Validation Overview

Cross-validation is performed using the script located at: scripts/crossval.py.

The basic usage is as follows:

python scripts/crossval.py [PATH TO CONFIG] --logdir [WHERE YOU WANT TO SAVE RESULTS]

The configs needed to produce the results (to be) reported in our paper are provided in the jobs/fugue/crossval directory. Each experiment will create a subdirectory of --logdir whose name corresponds to the experiment config, and the following information will be saved in seperate files:

  • debug.log: A log of all logging messages emitted during cross-validation.
  • cross_validation_results.csv: A csv file where each row corresponds to a hyperparameter configuration. The average and standard deviation of each evaluation metric over the k folds is provided.
  • best_config.yaml: The best hyperparameter configuration according to cross-validated dev performance.
  • best_model_scores.csv: The test (or dev-all in our case) performance of the best hyperparameter configuration. Performance is measured for multiple random seeds.

NOTE: The dataset paths in the configs are relative to the project root. If the datasets are stored elsewhere on your server, you probably will want to create a data/ symlink.

Running w/ SLURM

An example slurm shell script is provided at jobs/fugue/crossval/launch.sh. The #SBATCH directives are configured for the UCI NLP cluster, so to use this on another SLURM cluster you will need to make sure to update the nodelists, etc. The script allows for config files to be specified as an argument to facilitate queueing lots of jobs. For example you can run all of the albert-xxlarge MNLI experiments with:

for f in jobs/fugue/crossval/albert-xxlarge/mnli/*; do sbatch jobs/fugue/crossval/launch.sh $f; done

Generating the Results Table

Give the files in the results directory to Ivana or Eric (not Rob) to enter in the table :p

Clone this wiki locally