The default values for rope_freq_base and rope_freq_scale override the model values

[slaren]

The default values from these parameters are set to 10000 and 1:

llama-cpp-python/llama_cpp/llama.py

Lines 232 to 233 in a72efc7

	rope_freq_base: float = 10000.0,
	rope_freq_scale: float = 1.0,

To use the model default values, these should be set to zero instead.

This is the relevant code in llama.cpp:
https://github.com/ggerganov/llama.cpp/blob/2777a84be429401a2b7d33c2b6a4ada1f0776f1b/llama.cpp#L6699-L6701

Setting an incorrect value may result in poor generation quality in models that use a different value for these parameters, such as CodeLlama.

[m-from-space]

Thanks so much for just posting this, it fixed my issues with my models, if I set those parameters to 0.

[Ph0rk0z]

If I set 1 scale and 0 for base, it will still use 1.0 and 10,000.0? In models that have a specific value I make sure that it's what I want it.

I ask because some have metadata of 10,000 and some have it at 0.

[m-from-space]

If I set 1 scale and 0 for base, it will still use 1.0 and 10,000.0? In models that have a specific value I make sure that it's what I want it.

You have to use the latest llama-cpp-python (version 0.2.7) for those default values to work - set them to 0 to enforce defaults (not 0.0 though).

[slaren]

The metadata of the models should not be set to zero, it should either be the correct value or absent. If the model doesn't have the metadata, the default values are 10000 for rope_freq_base 1 for rope_freq_scale. If the conversion script is adding the metadata with value zero, that is likely a bug in llama.cpp.

[Ph0rk0z]

They were just converted before and/or from GGML. All the new ones auto populate with 10k where it's needed. In textgen scale tends to default to 1 and base tends to default to 0 unless the metadata is read.

I pulled this morning and yesterday, didn't want to be setting a base of literally 0 although that might have let itself be known rather quickly.

[Josh-XT]

This should be resolved by #767 in v0.2.8.