[Feature]: Add OpenAI server prompt_logprobs support

[Theodotus1243]

🚀 The feature, motivation and pitch

As noted in documentation OpenAI API don't support outputing only one token
But it's a very strong advantage over commertial models
Been able to get logits for prompt tokens

Alternatives

No response

Additional context

No response

[gabrielhuang]

Hey I'm doing the same thing, actually, you can try echo=True logprobs=1 and this should return the prompt logprobabilities. You have to disable prompt caching. You may have to use max_tokens=1 as well. Let me know if it works, and what parameters you use in the end.

[Theodotus1243]

@gabrielhuang Thanks, it's working
With vllm 0.5.2 (0.4.0post1 not working)

completion = client.completions.create(
        model=model_name,
        prompt=prompt,
        echo=True,
        logprobs=1,
        max_tokens=1,
    )

completion.choices[0].logprobs.tokens
completion.choices[0].logprobs.token_logprobs

Great workaround to switch to completions api

[DarkLight1337]

Reopening this as the workaround is not really ideal to solve this problem. It would be better to add an option to explicitly return the logprobs of the input prompt.

[cjfcsjt]

@DarkLight1337 According to the latest doc and sampling parameters

1. The chat API allows "echo", "logprobs", "max_tokens" as (extra) parameters, but actually it is not work for Vision LLM (e.g., I tried on served OpenGVLab/InternVL2-4B, which only returns output logprobs, not input.) The following code could reproduce this problem.

from openai import OpenAI
    client = OpenAI(
        base_url="http://localhost:8000/v1",
        api_key="token-abc123",
    )
    # sampling_params = SamplingParams(temperature=0.2, max_tokens=64, prompt_logprobs= 1, stop= ['<|eot_id|>'])
    completion = client.chat.completions.create(
        model="OpenGVLab/InternVL2-8B",
        messages= [
        {
            "role": "system",
            "content": "You are chatting with a language model."
        }
        ],
        extra_body={
            "stop": ['<|eot_id|>'],
            "echo": True,
            "max_tokens": 1,
            "logprobs": 1,
        }
    )

2. The the latest doc show that completions API does not allow "echo" as extra parameters, but actually it allows, as demonstrated by this issue, and the completions API can provide input prompt logprobs. However, the completions API seems only accept string as the input prompt, then how do we extend this to allow images?

[DarkLight1337]

1. The chat API allows "echo", "logprobs", "max_tokens" as (extra) parameters, but actually it is not work for Vision LLM (e.g., I tried on served OpenGVLab/InternVL2-4B, which only returns output logprobs, not input.) The following code could reproduce this problem.

There is currently no way to explicitly return logprobs for the input prompt in online inference, which is why I called the above solution a workaround (and reopened this issue). It would be great if this can be enabled similar to the offline SamplingParams.

2. The the latest doc show that completions API does not allow "echo" as extra parameters, but actually it allows, as demonstrated by this issue, and the completions API can provide input prompt logprobs. However, the completions API seems only accept string as the input prompt, then how do we extend this to allow images?

The Completions API was never made for image input. Since it is now considered legacy by OpenAI, we should focus on adding this feature to Chat Completions API instead.

[cjfcsjt]

There is currently no way to explicitly return logprobs for the input prompt in online inference, which is why I called the above solution a workaround (and reopened this issue). It would be great if this can be enabled similar to the offline SamplingParams.

Thanks for your patience. It would be great to see an option to return the logprobs of the input prompt (with image) in online inference.

[gnpinkert]

Could I take this issue since it has been reopened? It is unclear if anyone is working on it, so apologies if someone already is. @cjfcsjt @DarkLight1337

[DarkLight1337]

Could I take this issue since it has been reopened? It is unclear if anyone is working on it, so apologies if someone already is. @cjfcsjt @DarkLight1337

Not that I'm aware of. Thanks for helping out!

[gnpinkert]

Great, thanks! Will get right on it @DarkLight1337

[ashgold]

Hey I'm doing the same thing, actually, you can try echo=True logprobs=1 and this should return the prompt logprobabilities. You have to disable prompt caching. You may have to use max_tokens=1 as well. Let me know if it works, and what parameters you use in the end.

Hi @gabrielhuang ,
Why should I disable prompt caching?
if the prompt changed by even one character, the engine would only utilise the kv up to the block before that, so I thought I could enable prefix caching.