Adapter Support For the Longformer family

[gabinguo]

🚀 Feature request

Adapter Support For the Longformer models

Motivation

For question-answering over long documents, especially when the answers are long (surpass the 384 or 512 max sequence length supported by other LMs e.g. BERT, RoBERTa... It is impossible for users to get the correct answers...

So it seems necessary for us to use longformer or bigbird models which accept longer input sequences... however longer sequences will lead to a increase of fine-tuning computation..

I read the paper about adapter-transformer and I found this solution to be elegant and perfectly mitigate the problems of long-answer question answering.

Unfortunately, there is not yet a support in the adapter library the supports for longformer architecture. I would like to ask if you are planning a support for the longformer models?

Thanks : )

[gabinguo]

mentioned in #442

[logvinata]

I agree, this would be quit useful!