server : fix speculative decoding with context shift

[ggerganov]

fix #10547

Make sure the speculative batch does not exceed the slot's context.

[ngxson]

Do you think we should add a test case for this? Something like:

# test_speculative.py

def test_with_ctx_shift():
    global server
    server.n_ctx = 64
    server.start()
    res = server.make_request("POST", "/completion", data={
        "prompt": "Hello " * 64,
        "temperature": 0.0,
        "top_k": 1,
    })
    assert res.status_code == 200
    assert len(res.body["content"]) > 0

[ggerganov]

Yes, the error can be triggered with "prompt": "Hello " * 56, and then setting speculative.p_min = 0 in order to always generate a full draft batch of 16 tokens. This would cause to try to evaluate a speculative batch of 16 tokens when there is only 8 free tokens left in the context of the slot. With the changes in this PR, this should no longer fail.

[unclemusclez]

I've been running this PR for about an hour. it seems stable.

[josharian]

Works well for me. Thanks.