Long-Short Transformer for Long-Range Arena

This folder contains the source code for Long-Range Arena benchmark in Transformer-LS paper.

It is built on this repository

The implementation of long-short term attention is here.

Dataset Setup

To run the experiments, first, run the following to create the train, val and test data

cd datasets
python text.py
python listops.py
python retrieval.py

To download the datasets, one may refer to the instructions within Long-Range Arena repository

Scripts

Then, create the path for the checkpoints: mkdir LRA_chks. Finally, simply execute

bash run_text.sh

bash run_listops.sh

bash run_retrieval.sh

Reference repositories

• Nystromformer
• Long Range Arena