Add min p arg to server

[ArtyomZemlyak]

Add min p arg to server
Related to :
ggerganov/llama.cpp#3841

[iandennismiller]

It looks like min p was added to llama-cpp-python recently:

• in llama_cpp.py
• added approx. 2 weeks ago

[LorenzoBoccaccia]

I think it's missing from llama.py sampling

[ddh0]

Second this! It looks very promising and I'm excited.

[ddh0]

Pinging @abetlen, I finished this. See here: ddh0@e8e05bb

Apologies if this isn't the right way to submit a fix, I'm new to github. But I've tested it and it's working as expected. Let me know if I can help in any way.

[tk-master]

@ddh0 bad timing, I just made this pr :P #921

Testing would be appreciated.

[LorenzoBoccaccia]

tested at temperature 1.6 and 3.0 the assistant remains creative and repetition issues are gone even with a lower repetition penalty. at 3.0 seems to be too creative and basically ignore facts passed in the context, but remains coherent and inteligible.

still I think it should be disabled by default for backcompat

[m-from-space]

tested at temperature 1.6 and 3.0 the assistant remains creative and repetition issues are gone even with a lower repetition penalty. at 3.0 seems to be too creative and basically ignore facts passed in the context, but remains coherent and inteligible.

still I think it should be disabled by default for backcompat

Hey there, I just saw that this is now implemented. How exactly do I activate min_p sampling when calling the model? Is it activated by default?

[tk-master]

tested at temperature 1.6 and 3.0 the assistant remains creative and repetition issues are gone even with a lower repetition penalty. at 3.0 seems to be too creative and basically ignore facts passed in the context, but remains coherent and inteligible.
still I think it should be disabled by default for backcompat

Hey there, I just saw that this is now implemented. How exactly do I activate min_p sampling when calling the model? Is it activated by default?

the same way you'd change the temperature, top_k etc.. it is activated with a small value by default

[m-from-space]

the same way you'd change the temperature, top_k etc.. it is activated with a small value by default

Thank you. So in which order are sampling methods like min_p, top_k and top_p executed? Can I change that order?

[tk-master]

the same way you'd change the temperature, top_k etc.. it is activated with a small value by default

Thank you. So in which order are sampling methods like min_p, top_k and top_p executed? Can I change that order?

take a look at these lines here..

llama-cpp-python/llama_cpp/llama.py

Lines 1160 to 1168 in 4026166

	self._ctx.sample_top_k(candidates=self._candidates, k=top_k, min_keep=1)
	self._ctx.sample_tail_free(candidates=self._candidates, z=tfs_z, min_keep=1)
	self._ctx.sample_typical(
	candidates=self._candidates, p=typical_p, min_keep=1
	)
	self._ctx.sample_top_p(candidates=self._candidates, p=top_p, min_keep=1)
	self._ctx.sample_min_p(candidates=self._candidates, p=min_p, min_keep=1)
	self._ctx.sample_temp(candidates=self._candidates, temp=temp)
	id = self._ctx.sample_token(candidates=self._candidates)

The order is hardcoded at the moment.. but in my opinion you don't really want or need mixing min_p with top_p and top_k samplers, it's probably better to just disable top_p (top_p=1) etc and just use min_p and temp.
But if you wanna change the order and play with it, just edit that file.. it's simple.