[Feature]: Confidence score for Qwen/Qwen2-VL-7B-Instruct

[Dineshkumar-Anandan-ZS0367]

🚀 The feature, motivation and pitch

I have served a model Qwen/Qwen2-VL-2B-Instruct using vLLM.

I need to decode the confidence score for the model extracted output from the image using logprobs.

Is that is possible for this model. Give me some ideas on this. I got empty list logprobs results so the confidence is 0 for the correct answers also.

Alternatives

Reference from Huggingface

import vllm
import torch
import numpy as np
from vllm import LLM, SamplingParams

sampling_params = SamplingParams(
logprobs=20,
prompt_logprobs=20,
)

llm = LLM(model='/root/huggingface/math-shepherd-mistral-7b-prm', dtype='float32', tensor_parallel_size=1, gpu_memory_utilization=0.9, max_model_len=4096)
question = """Janet\u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?"""
output1 = """Step 1: Janet's ducks lay 16 eggs per day. ки\nStep 2: She eats three for breakfast every morning, so she has 16 - 3 = 13 eggs left. ки\nStep 3: She bakes muffins for her friends every day with four eggs, so she has 13 - 4 = 9 eggs left. ки\nStep 4: She sells the remainder at the farmers' market daily for $2 per fresh duck egg, so she makes 9 * $2 = $18 every day at the farmers' market. The answer is: 18 ки""" # 18 is right
output2 = """Step 1: Janet's ducks lay 16 eggs per day. ки\nStep 2: She eats three for breakfast every morning, so she has 16 - 3 = 13 eggs left. ки\nStep 3: She bakes muffins for her friends every day with four eggs, so she has 13 - 4 = 9 eggs left. ки\nStep 4: She sells the remainder at the farmers' market daily for $2 per fresh duck egg, so she makes 9 * $2 = $17 every day at the farmers' market. The answer is: 17 ки""" # 17 is wrong

good_token = '+'
bad_token = '-'
step_tag = 'ки'

tokenizer = llm.get_tokenizer()
[good_token_id, bad_token_id, step_tag_id] = tokenizer.encode(f"{good_token} {bad_token} {step_tag}")[1:] # [648, 387]

full_prompt = [f"{question} {output}" for output in [output1, output2]]
response = llm.generate(full_prompt, sampling_params)

for res in response:
prompt_logprobs = res.prompt_logprobs
all_tokens = res.prompt_token_ids
tag_token_index = [i for i, token in enumerate(all_tokens) if token == step_tag_id]
results = []
for token_index in tag_token_index:
logprobs = prompt_logprobs[token_index]
good_score = 0
bad_score = 0
if good_token_id in logprobs:
good_score = logprobs[good_token_id].logprob
if bad_token_id in logprobs:
bad_score = logprobs[bad_token_id].logprob

    normalized_good_score = torch.softmax(torch.tensor([good_score, bad_score]), dim=0)[0].item()
    results.append(normalized_good_score)
print(results)

Additional context

I have attached references

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

I suggest you set --task reward and use LLM.encode() to extract the hidden states directly from the model.

See here for more details.

[Dineshkumar-Anandan-ZS0367]

Currently the model supports (auto, generate, embedding) only. I am getting an error.

[DarkLight1337]

Can you update vLLM? It should be supported in the latest version.

[Dineshkumar-Anandan-ZS0367]

If i am specifying the score has a task, The model can give confidence score for the extracted data or only giving score

[DarkLight1337]

You can get the logits for each token (which is the hidden state of the model before passing to LM head).

[Dineshkumar-Anandan-ZS0367]

Thanks a lot sir, I will try it. Once i get the logits i will apply the torch softmax function and decode the score. Is correct?

[DarkLight1337]

Yes. Which task are you using specifically? You can also apply softmax automatically by overriding the pooler config (see --override-pooler-config argument).

[Dineshkumar-Anandan-ZS0367]

Can i set, Override or set the pooling method for pooling models. e.g. {“pooling_type”: “mean”, “normalize”: True}.’

normalize = False or True
pooling_type = "mean"

Correct?

[DarkLight1337]

If you want just the last token, you can use last-token pooling (assuming that there's no EOS token). Also, you should enable softmax, but not normalization if you want the score.

[Dineshkumar-Anandan-ZS0367]

You are asking like this or something else?

from vllm import LLM

Sample prompts.

text_1 = "What is the capital of France?"
texts_2 = [
"The capital of Brazil is Brasilia.", "The capital of France is Paris."
]

Create an LLM.

You should pass task="score" for cross-encoder models

model = LLM(
model="BAAI/bge-reranker-v2-m3",
task="score",
enforce_eager=True,
)

Generate scores. The output is a list of ScoringRequestOutputs.

outputs = model.score(text_1, texts_2)

Print the outputs.

for text_2, output in zip(texts_2, outputs):
score = output.outputs.score
print(f"Pair: {[text_1, text_2]!r} | Score: {score}")

Apologise, for the confusion sir.

I am performing extraction from the image. I have given prompt to extract all the key and value pair data. I have list of key and values. How to decode the confidence score for each keys. Because, I have 16Gb t4 machine. Could you share some snippet to help me on this.

[DarkLight1337]

Only cross-encoder models support the "score" task. I suggest you use the "classify" or "reward" task instead.

[DarkLight1337]

But, currently there is no easy way to get both the output and confidence score together.

You can try and follow #11397, which keeps the task as "generate" so you can get the text output in addition to the scores.

[Dineshkumar-Anandan-ZS0367]

Yes I am getting multiple errors. Are you integrate score or got logits from the qwen2-vl-7b model.
If you have please suggest and help me on this?

[DarkLight1337]

The best solution I have for you is #11397 as mentioned before.