Option to use ReBased Linear Attention and RingAttention

[kabachuha]

Not tested yet, as it needs training runs, but I think it can be helpful for you

ReBased repo: https://github.com/corl-team/rebased

Ring Attention implementation by Lucidrains https://github.com/lucidrains/ring-attention-pytorch

[maxin-cn]

Not tested yet, as it needs training runs, but I think it can be helpful for you

ReBased repo: https://github.com/corl-team/rebased

Thanks for your PR, we will check it.

[bhack]

Be aware that there could be some problem in the case you want to compile it
pytorch/pytorch#121386

[KimmiShi]

ring attention is not the most efficient method for this model.