Misc. bug: "Continue" action does not work correctly with grammars

[ScarletEmerald]

Name and Version

version: 7122 (21d31e0)
built with cc (Debian 14.2.0-19) 14.2.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server -m model.gguf -c 32768 -ngl 999 --reasoning-budget 0

Problem description & steps to reproduce

When using the new Continue button in the WebUI in combination with a grammar, continuing the generation restarts the grammar parsing from the point of continuation. What should happen is that the grammar parsing should start from the beginning of the original input.

For example, with the JSON grammar

{
  "json_schema": {
    "type": "object",
    "properties": {
      "sentiment": {
        "type": "string",
        "enum": ["positive", "neutral", "negative"]
      },
      "confidence": {
        "type": "number",
        "minimum": 0,
        "maximum": 1
      },
      "summary": {
        "type": "string",
        "maxLength": 200
      }
    },
    "required": ["sentiment", "confidence", "summary"]
  }
}

I say

How are you?

and get

{"sentiment": "positive", "confidence": 0.98, "summary": "The speaker seems to be in a good mood or is asking about the listener's well-being in a polite manner."}

If I delete after The speaker seems to and hit Continue, I get

{"sentiment": "positive", "confidence": 0.98, "summary": "The speaker seems to {"sentiment": "positive" , "confidence": 0.98, "summary": "The speaker seems to be asking a polite and casual greeting question to inquire about the listener's well-being." }

First Bad Commit

#16971

Relevant log output

[ScarletEmerald]

Come to think of it, this may be a difficult problem, since the user could theoretically edit the response into something that violates the grammar, and then try to continue generation. I wonder if that should throw an error, or if it should be the user's responsibility to avoid doing that.

[aldehir]

This is very challenging because both the grammar constraining and parsing start from generation, not from the start token in the prompt. It becomes even harder when every model is different, and finding a common approach seems impractical (to me at least).

[allozaur]

Come to think of it, this may be a difficult problem, since the user could theoretically edit the response into something that violates the grammar, and then try to continue generation. I wonder if that should throw an error, or if it should be the user's responsibility to avoid doing that.

Throwing an error due to grammar mistakes originating from tokens generated by models seems like an anti-pattern. And yes, exactly there can be cases where user manually edits the assistant response and hits continue.

This is very challenging because both the grammar constraining and parsing start from generation, not from the start token in the prompt. It becomes even harder when every model is different, and finding a common approach seems impractical (to me at least).

Exactly, models differ in terms of how they manage the "Continue" response, some do it incredibly good, others just semi-good.

I believe that this issue should be closed as non-planned due to the reasons described as above.