Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression

Code for Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC)

All experiments were run on TPUs using the Google TPU Research Cloud. Apply for TPU access at https://sites.research.google/trc/about/.

After provisioning a TPU VM, create a Python virtual environment using:

conda create -n icl -y python=3.10
conda activate icl

and install dependencies using

pip install -r requirements.txt
pip install -e .

To train a model, modify icl/configs/example.py and then run:

python run.py --config=icl/configs/example.py