Experiment Execution Challenge with RoFormer Model

[FedeVinc]

Hi,

I’ve been working with the RoFormer model recently and I'm specifically trying to train it using the Brassicales dataset. My goal is to compare its performance with a ConvNet model by running experiments on a GPU (configured using an sbatch file).

While everything works seamlessly with the ConvNet model, I’m encountering issues with RoFormer. Strangely, the debug phase runs without problems, but the training process fails when using the GPU.

I’ve attached the error details for reference. Have you had a chance to test the RoFormer model on a GPU, and if so, could you share any insights or configurations that worked for you?

I’m happy to provide additional details if needed. Thanks in advance for your help!

Best regards

[gonzalobenegas]

Hello, I'm sorry you are having issues. Here's a colab notebook training a small roformer for a few steps:
https://colab.research.google.com/drive/1c3EHJG19zMmVpPYyLBH7OTRXciHCEd3A?usp=sharing
Does that work for you?

I can also try help debug in your system. One note, I just updated the main branch of the repo with a small fix needed to run with the latest transformers library version.