Feat: Add rope scaling

[NanoCode012]

Closes #342

Note:

• This only applies it to llama arch. It should also support neox. Should I add specific if condition for this?

• It was stated on docs to NOT increase the max_position_embeddings ourselves for this. Let's say, if we want 8k, just change the rope factor.

• We leave validation to transformer.

• Test

[winglian]

lgtm

[winglian]

• It was stated on docs to NOT increase the max_position_embeddings ourselves for this. Let's say, if we want 8k, just change the rope factor.

has everyone else been doing it incorrectly? https://huggingface.co/togethercomputer/LLaMA-2-7B-32K/blob/main/config.json#L14

[NanoCode012]

• It was stated on docs to NOT increase the max_position_embeddings ourselves for this. Let's say, if we want 8k, just change the rope factor.

has everyone else been doing it incorrectly? https://huggingface.co/togethercomputer/LLaMA-2-7B-32K/blob/main/config.json#L14

It could be seen here on docs to not set it ourselves: https://github.com/gante/transformers/blob/30409af6e1b2b5efb6d9932b3e3b4ce20cfdb30e/src/transformers/models/llama/configuration_llama.py#L80-L87

At the same time, we also change this if the seq_len > the model's context, so I'm not sure if we should add a validation config check for that to make sure it does not happen.

Regarding the together's model, they have their own custom code for modelling_llama, since they require trust_remote_code. It's not exactly the same, so I can't say..

[NanoCode012]

This brought up another question in my mind, what about dataset packing? Do we just pass a 4k dataset to a "16k" dataset? If we do not set 16k seq_len, the dataset would be at most 4k? Do we need to add a condition to Not change max_pos_embedding if rope_scaling is true?

[NanoCode012]

Tested that it runs:

• with rope scaling
• with rope scaling , type: burp . Error as expected
• without rope scaling