Feature Request: fix handling of Qwen3-Embedding-0.6B input to add EOS token

[electroglyph]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

regarding the issue I posted here: #14234

Motivation

it's unintuitive to have to manually append the EOS token to inputs. it's not like this for transformers/onnx versions

Possible Implementation

No response

[ggerganov]

It would be helpful to get a summary about this EOS requirement across different embedding models:

• When do we add EOS token?
• When do we add CLS token?
• Do we add EOS token when pooling != last?
• etc.

I'm just not familiar enough with the embedding use-cases and don't know if there is some general logic about when to add these tokens.

[electroglyph]

it wouldn't be unreasonable for llama.cpp to require users to bring their own input_ids & attention_mask, that's how we do it in ONNX land

[ggerganov]

it wouldn't be unreasonable for llama.cpp to require users to bring their own input_ids

This is already supported - instead of string, simply pass an array of integers (the token ids).

[electroglyph]

i have a feeling it's linked to this section in tokenizer.json:

"post_processor": {
  "type": "Sequence",
  "processors": [
    {
      "type": "ByteLevel",
      "add_prefix_space": false,
      "trim_offsets": false,
      "use_regex": false
    },
    {
      "type": "TemplateProcessing",
      "single": [
        {
          "Sequence": {
            "id": "A",
            "type_id": 0
          }
        },
        {
          "SpecialToken": {
            "id": "<|endoftext|>",
            "type_id": 0
          }
        }
      ],
      "pair": [
        {
          "Sequence": {
            "id": "A",
            "type_id": 0
          }
        },
        {
          "Sequence": {
            "id": "B",
            "type_id": 0
          }
        },
        {
          "SpecialToken": {
            "id": "<|endoftext|>",
            "type_id": 0
          }
        }
      ],
      "special_tokens": {
        "<|endoftext|>": {
          "id": "<|endoftext|>",
          "ids": [
            151643
          ],
          "tokens": [
            "<|endoftext|>"
          ]
...

[CISC]

i have a feeling it's linked to this section in tokenizer.json:

Yes, indeed, we do not handle TemplateProcessing at all, but usually just rely on settings such as add_eos_token in tokenizer_config.json.

Maybe we can add some special-case checks for this, I'll look into it...

[CISC]

I see there are more complicated patterns than just (CLS) A (SEP) B (EOS), but for now I think we can get away with not implementing a full template.

[CISC]

BTW, they haven't added TemplaceProcessing (or 1_Pooling) to Qwen3-Reranker (yet), so it won't work.

[electroglyph]

i looked at the commit history for Qwen3-Embedding-0.6B, and they added the TemplateProcessing bit later as a fix...

[CISC]

i looked at the commit history for Qwen3-Embedding-0.6B, and they added the TemplateProcessing bit later as a fix...

Yes, but they forgot to fix the Reranker model. :)

Edit: I guess it's more complicated for the Reranker anyway as it requires a proper prompt.

[electroglyph]

yah, the official GGUFs of this embedding model are still broken too =/

thanks for the fix!

[CISC]

yah, the official GGUFs of this embedding model are still broken too =/

Yes, unfortunately they will have to be reconverted/edited to get the appropriate add_eos_token metadata.