bug: Duplicate BOS Token in Hugging Face Chat Templates

[Van-QA]

Description:
When using chat templates in Hugging Face, the Beginning-OfSentence (BOS) token is often already included in the template. However, Llama.cpp also automatically adds the BOS token, resulting in a duplicate BOS token.

Expected behavior:
The system should automatically detect and remove any duplicate BOS tokens in the chat template. This would ensure proper functioning of the chat system without causing errors due to redundant tokens.

Additional context:
This issue may cause unexpected behavior or errors in the chat system. It is recommended that Cortex checks for and deduplicates the BOS token if it is present in the user's template to maintain a consistent and error-free chat experience.

[dan-menlo]

@nguyenhoangthuan99 I am putting this as a sub-issue of #1151. This issue may be stale if HF has fixed upstream

[nguyenhoangthuan99]

Problem
This is example of huggingface chat template that we are using <|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n

The token <|begin_of_text|> is BOS token. However, when we use this chat template, the llama.cpp engine automatically add <|begin_of_text|> to the prompt -> there are 2 BOS token in the prompt e.g. <|begin_of_text|><|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{system_message}<|eot_id|>.

This issue won't affect too much to the response, and most models still work well with this. We are using cortex with double BOS token with llama.cpp engine without any error.

Solution
There are 2 solutions right now for this:

• Handle at cortex.llamacpp engine. Every time, when run inference we need to check if there is BOS token of that model in the chat template. the BOS token is parsed from GGUF file by llama.cpp
• Handle at cortex.cpp side. We need to save BOS token to model.yml and let cortex.cpp remove it when request to cortex.llamacpp engine.

I think the first solution is better because we don't need to update the cortex.cpp part and model.yml file which is related to many other parts of cortex.cpp.

This issue is not critical, and it needs effort to read llama.cpp source to handle.

[nguyenhoangthuan99]

I want to close this issue because llama.cpp set tokenizer_add_bos default to false on this PR

For that reason, this bug no longer appear when running inference with llama.cpp.
Tested with cortex.cpp, when running inference, no warning log about duplicated BOS token appeared.

[gabrielle-ong]

Solved by llama.cpp upstream (thanks @nguyenhoangthuan99 for investigating)