Skip to content

[Bugfix] : Prompt logprobs clean #13406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[Bugfix] : Prompt logprobs clean #13406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bffb799
Fix docstring prompt_logprobs default value
maxDavid40 Feb 17, 2025
ee09046
create static method to clean prompt_logprobs
maxDavid40 Feb 17, 2025
ba195f5
use static method to clean prompt logprobs
maxDavid40 Feb 17, 2025
334a6f1
Fix docstring mistake
maxDavid40 Feb 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions vllm/entrypoints/openai/serving_chat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -841,6 +841,8 @@ async def chat_completion_full_generator(

request_metadata.final_usage_info = usage

self._clean_prompt_logprobs(final_res.prompt_logprobs)

response = ChatCompletionResponse(
id=request_id,
created=created_time,
Expand Down
8 changes: 2 additions & 6 deletions vllm/entrypoints/openai/serving_completion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,13 +396,9 @@ def request_output_to_completion_response(
for final_res in final_res_batch:
prompt_token_ids = final_res.prompt_token_ids
assert prompt_token_ids is not None
self._clean_prompt_logprobs(final_res.prompt_logprobs)
prompt_logprobs = final_res.prompt_logprobs
if prompt_logprobs:
for logprob_dict in prompt_logprobs:
if logprob_dict:
for logprob_values in logprob_dict.values():
if logprob_values.logprob == float('-inf'):
logprob_values.logprob = -9999.0

prompt_text = final_res.prompt

token_ids: GenericSequence[int]
Expand Down
14 changes: 14 additions & 0 deletions vllm/entrypoints/openai/serving_engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -499,6 +499,20 @@ async def _get_trace_headers(

return None

@staticmethod
def _clean_prompt_logprobs(
prompt_logprobs: Union[None, List[Union[None, Dict]]]) -> None:
"""
Preprocess prompt_logprobs result
to avoid starlette.response.JSONResponse's error
"""
if prompt_logprobs:
for logprob_dict in prompt_logprobs:
if logprob_dict:
for logprob_values in logprob_dict.values():
if logprob_values.logprob == float('-inf'):
logprob_values.logprob = -9999.0

@staticmethod
def _base_request_id(raw_request: Optional[Request],
default: Optional[str] = None) -> Optional[str]:
Expand Down