Update Chat Template #71

thameem-abbas · 2025-01-07T16:31:04Z

This PR is to update the default chat template (to prompt only) and introduce support for custom prompt formats.

vLLM uses tokenizer provided chat templates in the /v1/chat/completions endpoint.
vLLM does not add anything in the /v1/completions endpoint and allows us to precisely control/filter the sequences sent in the request. This PR now allows us to also use custom prompts via the "custom_prompt_format" field.

This PR fixes #67

…ormatting.

…nly.

dataset.py

dagrayvid

Looks good to me now!

thameem-abbas added 4 commits January 2, 2025 13:15

Fix: Change default behavior of llm-load-test to avoid using prompt f…

d7c779a

…ormatting.

ADD: Custom Prompt Format option for unsupported model families

1af2be5

FIX: Update default prompt to match vLLM's expected input of prompt o…

fa99728

…nly.

FIX: Make behavior more consistent and move changes to single function.

6256486

dagrayvid reviewed Jan 7, 2025

View reviewed changes

dataset.py Outdated Show resolved Hide resolved

FIX: Remove get_format_string function

a6bac69

thameem-abbas self-assigned this Jan 7, 2025

thameem-abbas added 2 commits January 7, 2025 17:29

Merge branch 'openshift-psap:main' into dev/update-chat-template

980a611

FIX: Remove unused import - warnings module

4a373d1

thameem-abbas requested a review from dagrayvid January 8, 2025 18:29

dagrayvid approved these changes Jan 8, 2025

View reviewed changes

dagrayvid merged commit 6e43ce8 into openshift-psap:main Jan 8, 2025
1 check passed

thameem-abbas deleted the dev/update-chat-template branch January 8, 2025 19:19

Provide feedback