Fix OpenAI ChatCompletion Ignore stop from FastChat Conv Template #1503
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I replaced the implementation of this function already in: #1493 That should be the way to solve this anyways. FastChat is a hack for formatting. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using vLLM's OpenAI API server to serve models, I find that the ChatCompletion request by default does not honor the
stop_token_idsandstop_strset by FastChat conversation templates. It will cause issues (model keep generating irrelevant stuff) when using vLLM served OpenAI API as an OpenAI compatible server for Gradio interface of FastChat.This PR add a check in OpenAI ChatCompletion request to make sure the
stop_token_idsandstop_strare merged with the request before send it to generation, similar to the implementation of FastChatvllm_worker.