Increasing Context Length of Pretrained Models

[rsomani95]

Most of the pretrained models available have a context length of 77.
(There's a few exceptions, like the mt5 models discussed here)

Is there a way to fine-tune a model that's got an existing context length of 77 with a larger context?

Say I want to increase the context length to 200. Intuitively, I can think of two things to try:

1. Set the .positional_embedding to be the same as the pretrained model, and start training with random init for positions 78-200
2. Interpolate the .positional_embedding from existing context length to the desired context length

Do either of those make sense; I'm curious if anyone's tried something like this and what would be a good starting point for experimentation?