Baichuan2-7B-Chat model converted to ggml-model-q4_0.gguf, AI answer does not stop automatically when inference is made

[Lyzin]

I converted the Baichuan2-7B-Chat model to ggml-model-q4_0.gguf and then used . /main inference and found that I could not stop answering

my system: macos
python version: 3.9.10

Here's my version using llama.cpp
. /main version：

llama.cpp git commit id:

Here are the steps I took to convert

python convert-hf-to-gguf.py ./models/Baichuan2-7B-Chat

./quantize ./models/Baichuan2-7B-Chat/ggml-model-f16.gguf ./models/Baichuan2-7B-Chat/ggml-model-q4_0.gguf q4_0

Here's how to reason with . /main reasoning results

./main -m ./models/Baichuan2-7B-Chat/ggml-model-q4_0.gguf -n 256 --repeat_penalty 1.0 -ngl 0 --color -i -r "User:" -f prompts/chat-with-baichuan.txt

Here is the result of the inference, which keeps repeating the output

Is this a problem caused by the conversion?

[hiepxanh]

@Lyzin I belive it is the same issue with this #3969
can you download Phi-2 model and confirm the bug still happen?

[Lyzin]

@Lyzin I belive it is the same issue with this #3969 can you download Phi-2 model and confirm the bug still happen?

I downloaded the phi-2 model, requantized it, and the output automatically stopped
llama.cpp version tag: b1966

./main -m ./models/phi-2/ggml-model-q4_0.gguf -n 512 --color -i -cml -ngl 0 -r "User:" -f prompts/chat-with-bob.txt

Here is the output of the AI

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.