I might attempt this one next since we use it in production, unless anyone else wants to pick it up?

I know the T5 attempt was pretty unwieldy/slow on my part, but I think I learned a fair bit about the codebase and it should hopefully go smoother next time.

It would be interesting to see if one could get the adapters to interact with the sparse attention mechanism non-trivially, but that would be ambitious.

Hey @AmirAktify, would be great if you could implement this, there's no one from our side currently working on it.

Also, thanks again for your help on the T5 implementation which is finally released now and where you did most of the work. We're always happy for any help we can get from the community, therefore really appreciating your contributions 😃

Any progress on this? I can probably help with implementation.