Can I directly obtain the logits here?

[SparkJiao]

Hi, wonderful work!

I want to know if there is a easy way to obtain the logits, since sometimes I only need to calculate the perplexity/language modeling loss of specific sequence.

I saw the code here: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py#L211-L235

So I want to know if I directly use the logits generated by lm_head, can I be benefited from the paged attention framework? Thanks very much!

[zhuohan123]

For sequence generation, you can get the logits of the generated tokens by setting log_probs option in SamplingParams.

In the current vLLM, we haven't added support for getting the logits of the prompt parts yet. Theoretically, your specific use case can benefit from the no-padding batching of different-length sequences in vLLM, so vLLM can still be faster than plain huggingface transformers. However, since you only compute on prompts, there is no "KV Cache" and thus the memory management part of PagedAttention does not increase throughput.

[SparkJiao]

I see. Thansk for your clarification very much! I will try some cases first

[jpiabrantes]

@zhuohan123

For sequence generation, you can get the logits of the generated tokens by setting log_probs option in SamplingParams.

logits are different from logprobs. log_probs = log(exp(logits)/sum(exp(logits)))

We can't recover the top 10 logits from the top 10 logprobs. Can this issue be re-open?

[junzhang-zj]

can i obtain the logits of the prompt, like the loss of evaluation？

[vmoens]

Is there any update on this? Has anyone figured out how to retrieve logits for a given input?

[Dineshkumar-Anandan-ZS0367]

Is there any option to get score or logits from the model using vLLM: Qwen/Qwen2-VL-7B-Instruct

I need to get the confidence score for the data extracted from image

[athmanar]

logprobs and logits are different things. Is there any update on this?

[metasyn]

@zhuohan123 / @SparkJiao could you review the various comments here, discussing how this was maybe closed prematurely? i think we need to re-open this 🙇