Transformers-tutorials

Hi there folks!

This repository contains demo(s) to the models I ported to the Transformers library by 🤗 HuggingFace. Currently, all of them are implemented in PyTorch.

NOTE: This readme has been highly inspired from @NielsRogge transformers-tutorials repo.

NOTE: if you are not familiar with HuggingFace and/or Transformers, I highly recommend to check out the free course they provide, which introduces you to several Transformer architectures (such as BERT, GPT-2, T5, BART, etc.), as well as an overview of the HuggingFace libraries, including Transformers, Tokenizers, Datasets, Accelerate and the hub.

For an overview of the ecosystem of HuggingFace for computer vision (June 2022), refer to this notebook with corresponding video.

Currently, it contains the following demos:

• Transformers are Sample-Efficient World Models (IRIS) (paper):
  • train IrisModel on Atari environment(s) to play breakout or any environment id provided.

Training frameworks

Regarding fine-tuning Transformer models (or more generally, PyTorch models), there are a few options:

• HuggingFace Trainer. The HuggingFace Trainer API can be seen as a framework similar to PyTorch Lightning in the sense that it also abstracts the training away using a Trainer object. However, contrary to PyTorch Lightning, it is not meant not be a general framework. Rather, it is made especially for fine-tuning Transformer-based models available in the HuggingFace Transformers library. The Trainer also has an extension called Seq2SeqTrainer for encoder-decoder models, such as BART, T5 and the EncoderDecoderModel classes. Note that all PyTorch example scripts of the Transformers library make use of the Trainer.
• HuggingFace Accelerate: Accelerate is a new project, that is made for people who still want to write their own training loop , but would like to make it work automatically irregardless of the hardware (i.e. multiple GPUs, TPU pods, mixed precision, etc.).