Rotary Embeddings in Encoder

[conceptofmind]

Hi @Sengxian,

I was wondering how you handled issues relating to length extrapolation for Rotary Embeddings and relative positional encoding for the encoder? From my understanding, Rotary Embeddings only work properly for the decoder since they do not length-generalize that well.

Thank you,

Enrico