bump to vllm0.6.2 and add explicit chat template #3964

hustxiayang · 2024-09-27T15:23:35Z

What this PR does / why we need it:
Bump to vLLM0.6.2 to support multimodal models.
vLLM0.6.2 requires newer version of transformers '>=4.45.0`, some related changes are:

It does not support default chat templates for models: 🚨 No more default chat templates huggingface/transformers#31733.
Thus, add an optional chat template member in CreateChatCompletionRequest class, which is similar to https://github.com/vllm-project/vllm/blob/9aaf14c62e16a7c74b5192a44d01a78125dab2fc/vllm/entrypoints/openai/protocol.py#L239
In tokenizer, clean_up_tokenization_spaces would be set as false by default, check details: [BUG] GPT-2 tokenizer is NOT invertible huggingface/transformers#31884. Thus, change the assertions in test test_input_padding_with_pad_token_not_specified

Also added a simple unit test of chat_template for tokenizer.

Type of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

add explicit chat template
clean_up_tokenization_spaces were set as False by transformers

hustxiayang · 2024-09-27T16:30:55Z

/rerun-all

hustxiayang · 2024-09-30T16:45:42Z

With newer versions of transformers, it does not support default chat templates for models: huggingface/transformers#31733, thus I explicitly give a chat template in tests

terrytangyuan

Can you update the PR title and description to also include any features that you'd like to bring in this release?

hustxiayang · 2024-09-30T19:31:15Z

@terrytangyuan thanks a lot for your reply. I've updated the title and description!

spolti

The changes look good to me, however, my only concern is backward compatibility.
For users migrating to the new KServe version which brings it, will user be required to do anything else or?

hustxiayang · 2024-10-01T15:42:40Z

@spolti I thought the added parameter has a default value None, so it would not affect the users? If it's not set, the model's tokenizer should have some chat templates, which behave the same as tokenizer's apply_chat_template method? but I will double check it.

johnugeorge · 2024-10-06T05:43:06Z

We should have 0.14.0 stable release cut while we thoroughly test these new change separately.

hustxiayang · 2024-10-07T05:06:33Z

We should have 0.14.0 stable release cut while we thoroughly test these new change separately.

@johnugeorge Hi, thanks for your comment! Do you mean that I should expect some more PRs on 0.14.0 stable release beyond #3970?

Signed-off-by: yxia216 <yxia216@bloomberg.net>

yuzisun

/lgtm
/approve

yuzisun · 2024-10-16T11:19:10Z

Thanks @hustxiayang ! great work on this

* explicitly give a chat template Signed-off-by: yxia216 <yxia216@bloomberg.net> * fix dummy model issue, fix python version smaller than 3.10, and formatting Signed-off-by: yxia216 <yxia216@bloomberg.net> * fix vLLMModel Signed-off-by: yxia216 <yxia216@bloomberg.net> * change the interface of CreateChatCompletionRequest Signed-off-by: yxia216 <yxia216@bloomberg.net> * update dummy model's para Signed-off-by: yxia216 <yxia216@bloomberg.net> * consitent with OpenAIGPTTokenizer and OpenAIGPTModel Signed-off-by: yxia216 <yxia216@bloomberg.net> * give a chat template if there is no Signed-off-by: yxia216 <yxia216@bloomberg.net> * update the response and update the readme Signed-off-by: yxia216 <yxia216@bloomberg.net> * update the chat_template Signed-off-by: yxia216 <yxia216@bloomberg.net> * update data Signed-off-by: yxia216 <yxia216@bloomberg.net> * add test of chat temmplate for tokenizer Signed-off-by: yxia216 <yxia216@bloomberg.net> * jinja2 template format Signed-off-by: yxia216 <yxia216@bloomberg.net> * use a simpler chat template --------- Signed-off-by: yxia216 <yxia216@bloomberg.net> Signed-off-by: Snehomoy <snehomoy6@gmail.com>

* add tags to rest server timing logs to differentiate cpu and wall time (kserve#3954) Signed-off-by: Gregory Keith <gregory.keith@maxkelsen.com> * Implement Huggingface model download in storage initializer (kserve#3584) * initial commit for hugging face model download and load Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * bug fix on storage initializer Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * added hf_token and unittests Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * separate hf-storage-initializer image to reduce image size Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * review comment changes Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * snapshot download Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * use existing image for storage initializer Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * resolved merge conflicts Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * added hf storage uri validation Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * resolved merge conflicts Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> --------- Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * Update OWNERS file (kserve#3966) Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Cluster local model controller (kserve#3860) * Consolidate into one commit Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix configmap format Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix configmap Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Log configmap read error Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * fix naming Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Update comments Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add enabled flag to configmap and avoid cluster resource check in isvc defaulter Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * move client into the local model block Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Fix lint Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> * Prepare for 0.14.0-rc1release and automate sync process (kserve#3970) * Sync helm chart with kustomize Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update manifest generation script to sync helm charts Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Make kserve-addressable-resolver role optional Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Prepare for 0.14.0-rc1 release Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update release process Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Comment out crd sync script in make Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix helm template syntax Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> --------- Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * add a new API for multi-node/multi-gpu (kserve#3871) * add a new API for multi-node/multi-gpu Signed-off-by: jooho lee <jlee@redhat.com> * fix gitaction Signed-off-by: jooho lee <jlee@redhat.com> * fix merging conflict Signed-off-by: jooho lee <jlee@redhat.com> * fix gitaction fail Signed-off-by: jooho lee <jlee@redhat.com> * regenerate codegen/manifests Signed-off-by: jooho lee <jlee@redhat.com> * Apply suggestions from code review Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Jooho Lee <ljhiyh@gmail.com> * remove unnecessary comment Signed-off-by: jooho lee <jlee@redhat.com> * change the type of workerSpec in isvc to PodSpec Signed-off-by: jooho lee <jlee@redhat.com> * update controller-gen version Signed-off-by: jooho lee <jlee@redhat.com> * remove replicas from workerSpec Signed-off-by: jooho lee <jlee@redhat.com> * fix conflict merging Signed-off-by: jooho lee <jlee@redhat.com> * added size(replicas) for workerSpec again Signed-off-by: jooho lee <jlee@redhat.com> * add WorkerSpec to inferenceService Signed-off-by: jooho lee <jlee@redhat.com> * fix go linter Signed-off-by: jooho lee <jlee@redhat.com> --------- Signed-off-by: jooho lee <jlee@redhat.com> Signed-off-by: Jooho Lee <jlee@redhat.com> Signed-off-by: Jooho Lee <ljhiyh@gmail.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net> * Fix update-openapigen.sh that can be executed from kserve dir (kserve#3924) * fix openapigen.sh that can be executed from kserve dir Signed-off-by: jooho lee <jlee@redhat.com> * regenerate codegen/manifests Signed-off-by: jooho lee <jlee@redhat.com> * Update go.sum Signed-off-by: Dan Sun <dsun20@bloomberg.net> --------- Signed-off-by: jooho lee <jlee@redhat.com> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net> * Add python 3.12 support and remove python 3.8 support (kserve#3645) * Support python 3.12 Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update dependencies Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update deps to support 3.12 Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Remove python 3.8 support Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Remove skip for infer client test Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix port forward Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix sklearn pandas dep Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * skip pydantic v1 test for py 3.12 Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add setuptools dep for pmml Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix lgb Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Include setuptools for paddle Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Include setuptools for huggingface Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Rebase Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Rebase Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> --------- Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix openssl vulnerability CWE-1395 (kserve#3975) Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix Kubernetes Doc Links (kserve#3670) * Bump version to 0.13.0-rc0 (kserve#3665) Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> Signed-off-by: jordanyono <jordanyono@gmail.com> * fixing docs Signed-off-by: jordanyono <jordanyono@gmail.com> * fix spelling mistake Signed-off-by: jordanyono <jordanyono@gmail.com> --------- Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> Signed-off-by: jordanyono <jordanyono@gmail.com> Co-authored-by: Curtis Maddalozzo <cmaddalozzo@users.noreply.github.com> * Fix kserve local testing env (kserve#3981) * Fix local testing Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Fix codegen Signed-off-by: Dan Sun <dsun20@bloomberg.net> --------- Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Fix streaming response not working properly with logger (kserve#3847) Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add a flag for automount serviceaccount token (kserve#3979) * Add a flag for automount serviceaccount Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Set default to false Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Default to true Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix test error Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Update openapi generated.go Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix python lint Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix config loading Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Do not set security context on the storage initializer from user container (kserve#3985) * Do not set security context on the storage initializer from user container Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add securityContext to the default storage container in the helm chart Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Modelcar race condition mitigation with an init container (kserve#3932) This adds the model container as an init-container to mitigate a race condition that would happen if the model container is not present on the cluster-node. The race condition happens if the cluster is able to fetch and start the runtime container before the modelcar is fetched. This would lead to the runtime to terminate with error. By configuring the model container as an init-container the runtime won't start until the modelcar is fetched. Although there is still the risk of a race condition when the cluster schedules the runtime container first, the pod should stabilize after a few restarts of the runtime container and should either prevent a CrashLoopBackOff event on the pod, or the crash event would finish quickly. This improves compatibility with the runtimes which can now stay agnostic to the modelcar implementation, until better techniques (like native sidecars, and oci volume mounts) become mature. Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com> * Fix: Headers passing for v1/v2 endpoints (kserve#3669) * Initial commit for headers passing issue Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * modifying the e2e test for rebase conflict Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * bug fix on unittest Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * review changes Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * fix for test failure Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * bug fix on e2e test Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * overridding the entrypoint of custom model images Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * custom response header Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * fix for unittest failure Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * added custom response headers in post process Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * added predict time latency in example response header Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * fix OOM --------- Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net> * Torchserve security update (kserve#3774) * security update Signed-off-by: udai <udaij12@gmail.com> * adding sign off Signed-off-by: udai <udaij12@gmail.com> --------- Signed-off-by: udai <udaij12@gmail.com> * Pin ubuntu 22.04 for minikube setup action (kserve#3994) Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * KServe 0.14 Release (kserve#3988) * temp commit Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * python-release.sh Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * bump to vllm0.6.2 add explicit chat template (kserve#3964) * explicitly give a chat template Signed-off-by: yxia216 <yxia216@bloomberg.net> * fix dummy model issue, fix python version smaller than 3.10, and formatting Signed-off-by: yxia216 <yxia216@bloomberg.net> * fix vLLMModel Signed-off-by: yxia216 <yxia216@bloomberg.net> * change the interface of CreateChatCompletionRequest Signed-off-by: yxia216 <yxia216@bloomberg.net> * update dummy model's para Signed-off-by: yxia216 <yxia216@bloomberg.net> * consitent with OpenAIGPTTokenizer and OpenAIGPTModel Signed-off-by: yxia216 <yxia216@bloomberg.net> * give a chat template if there is no Signed-off-by: yxia216 <yxia216@bloomberg.net> * update the response and update the readme Signed-off-by: yxia216 <yxia216@bloomberg.net> * update the chat_template Signed-off-by: yxia216 <yxia216@bloomberg.net> * update data Signed-off-by: yxia216 <yxia216@bloomberg.net> * add test of chat temmplate for tokenizer Signed-off-by: yxia216 <yxia216@bloomberg.net> * jinja2 template format Signed-off-by: yxia216 <yxia216@bloomberg.net> * use a simpler chat template --------- Signed-off-by: yxia216 <yxia216@bloomberg.net> * bump to vllm0.6.3 (kserve#4001) Signed-off-by: yxia216 <yxia216@bloomberg.net> * Feature: Add hf transfer (kserve#4000) * Add hf transfer Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com> * Add hf transfer env Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com> --------- Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com> * Fix snyk scan null error (kserve#3974) Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update quick install script (kserve#4005) Signed-off-by: Johnu George <johnugeorge109@gmail.com> * Local Model Node CR (kserve#3978) * init CR Signed-off-by: Gavin Li <yli1109@bloomberg.net> * make generate Signed-off-by: Gavin Li <yli1109@bloomberg.net> * make manifests Signed-off-by: Gavin Li <yli1109@bloomberg.net> * black format Signed-off-by: Gavin Li <yli1109@bloomberg.net> * fix generated python code Signed-off-by: Gavin Li <yli1109@bloomberg.net> * feedback Signed-off-by: Gavin Li <yli1109@bloomberg.net> * more feedback Signed-off-by: Gavin Li <yli1109@bloomberg.net> * black format Signed-off-by: Gavin Li <yli1109@bloomberg.net> * make manifests Signed-off-by: Gavin Li <yli1109@bloomberg.net> --------- Signed-off-by: Gavin Li <yli1109@bloomberg.net> * Reduce E2Es dependency on CI environment (2) (kserve#4008) Reduce E2Es dependency on CI environment Some code of the E2Es assume the environment is GitHub, because it is referring to GitHub-specific variables. This PR focuses on the `kserve/custom-model-grpc` container image, so that no Python code of the E2Es using this image is referencing the `github_sha` variable. Also, a small improvement on the `get_isvc_endpoint` utility function is done to use the schema in the endpoint specified in the status of the InferenceService, rather than hard-coding to plain-text HTTP. This adds compatibility for CI environments where KServe ConfigMap has been configured with `urlScheme: https` for the Ingress. Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com> * Allow GCS to download single file (kserve#4015) allow gcs to download single file fixes kserve#4013 Signed-off-by: Spolti <fspolti@redhat.com> * bump to vllm0.6.3.post1 (kserve#4023) Signed-off-by: yxia216 <yxia216@bloomberg.net> * Set default for SamplingParams.max_tokens in OpenAI requests if unset (kserve#4020) * Set default for SamplingParams.max_tokens in OpenAI requests if unset Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com> * Fix lint Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com> * Fix formatting Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com> --------- Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com> * Add tools functionality to vLLM (kserve#4033) * Add tools to chat template Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Linting Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> add test Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Fix linting manually Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> * Fix linting Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> --------- Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Signed-off-by: Arjun Bhalla <arjunbhallasg@gmail.com> Co-authored-by: Arjun Bhalla <abhalla31@bloomberg.net> * Use apt-get upgrade for CVE fixes Signed-off-by: Dan Sun <dsun20@bloomberg.net> * For vllm users, our parser should be able to support both - and _ (kserve#3933) Signed-off-by: yxia216 <yxia216@bloomberg.net> * Add tools unpacking for vLLM (kserve#4035) * Add tools to chat template Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Linting Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> add test Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Fix linting manually Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> * Fix linting Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> * Add tools unpacking for vllm Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> * Add sanity check test Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> --------- Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Signed-off-by: Arjun Bhalla <arjunbhallasg@gmail.com> Co-authored-by: Arjun Bhalla <abhalla31@bloomberg.net> * Multi-Node Inference Implementation (kserve#3972) Signed-off-by: jooho lee <jlee@redhat.com> * Enhance InjectAgent to Handle Only HTTPGet, TCP Readiness Probes (kserve#4012) * Fix readiness probe logic and update test scenarios for HTTPGet, TCPSocket, and Exec handling Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Update: Refactor logic for readiness probe handling Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Apply gofmt formatting to agent_injector.go Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Added logger to replace fmt.Printf for better consistency and observability Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Formatted file using goimports with -local Signed-off-by: Snehomoy <snehomoy6@gmail.com> --------- Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Feat: Fix memory issue by replacing io.ReadAll with io.Copy (kserve#4017) (kserve#4018) * Feat: Fix memory issue by replacing io.ReadAll with io.Copy (kserve#4017) Previously, io.ReadAll was causing out-of-memory problems when downloading large files from GCS. This change replaces io.ReadAll() with io.Copy() to stream data and prevent excessive memory usage. Signed-off-by: ops-jaeha <me@jaehai.com> * Feat: Fix add newline at end of file to satisfy golang lint Signed-off-by: ops-jaeha <me@jaehai.com> * Feat: Refact log Info for golang lint (kserve#4017) Signed-off-by: ops-jaeha <me@jaehai.com> --------- Signed-off-by: ops-jaeha <me@jaehai.com> * Update alibiexplainer example (kserve#4004) chore: Fix CVE-2024-26130 - NULL Pointer Dereference - Upgrade cryptography to version 42.0.4 or higher. Update Python version to match KServe 0.14.0 Update tensorflow, tensorflow-io-gcs-filesystem and dill libraries Signed-off-by: Spolti <fspolti@redhat.com> * Fix huggingface build runs out of storage in CI (kserve#4044) Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update snyk scan to include new images (kserve#4042) Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Introducing KServe Guru on Gurubase.io (kserve#4038) Signed-off-by: Kursat Aktas <kursat.ce@gmail.com> * Fix Hugging Face server EncoderModel not returning probabilities (kserve#4024) * Fix huggingface srever not work with return_probabilities Signed-off-by: oplushappy <ken910304@gmail.com> * Fix pytest huggingface server assertion error Signed-off-by: oplushappy <ken910304@gmail.com> * Fix the lint error and Add approx for assertion Signed-off-by: oplushappy <ken910304@gmail.com> * Parse string output to dictionary for accurate assertion Signed-off-by: oplushappy <ken910304@gmail.com> * Fix linting error Signed-off-by: oplushappy <ken910304@gmail.com> --------- Signed-off-by: oplushappy <ken910304@gmail.com> * Add deeper readiness check for transformer (kserve#3348) * Add deeper readiness and liveness check for transformer Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add unit tests Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * put the feature behind flag Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update tests Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * resolve comments Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Make use of inference client Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add e2e test Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Make inference client singleton and lazy initialize Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Raise 503 If server is not ready / live Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add test for custom transformer with rest protocol Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix CI running out of space Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Increase memory limit Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Check for model ready Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Webhook debug Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Address reviews Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Check for retry count in grpc client Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update python/kserve/kserve/model_server.py Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Sivanantham <90966311+sivanantha321@users.noreply.github.com> --------- Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> Signed-off-by: Sivanantham <90966311+sivanantha321@users.noreply.github.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net> * Fix Starlette Denial of service (DoS) via multipart/form-data (kserve#4006) chore: Fix CVE-2024-47874 Signed-off-by: Spolti <fspolti@redhat.com> * remove duplicated import "github.com/onsi/gomega" (kserve#4051) remove duplicated import Signed-off-by: carlory <baofa.fan@daocloud.io> * Fix localmodel controller name in snyk scan workflow (kserve#4054) Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix azure blob storage access key env not mounted (kserve#4064) * add storageaccesskey to azure env builder Signed-off-by: bentohset <toh.benjamin123@gmail.com> * update integration and unit test for azure storage access key Signed-off-by: bentohset <toh.benjamin123@gmail.com> * fix formatting Signed-off-by: bentohset <toh.benjamin123@gmail.com> --------- Signed-off-by: bentohset <toh.benjamin123@gmail.com> * Storage Initializer support single digit azure DNS zone ID (kserve#4070) * support single digit azure zone id Signed-off-by: bentohset <toh.benjamin123@gmail.com> * add single digit azure dns zone id tests Signed-off-by: bentohset <toh.benjamin123@gmail.com> * fix formatting Signed-off-by: bentohset <toh.benjamin123@gmail.com> --------- Signed-off-by: bentohset <toh.benjamin123@gmail.com> * support text embedding task in huggingfaceserver Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> * fix lint errors Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> * format code Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> * bring back enhancements after getting kserve up-to-date (#42) * improve dockerfile, makefile, readme * support custom classification labels, refactor postprocess * support text embedding task * improve support for token classification (named entity recognition) * use self.model_config.id2label by default (#45) * minor cleanup and fixes after rebase * use approx in test_input_padding * revert token_classification changes * fix test --------- Signed-off-by: Gregory Keith <gregory.keith@maxkelsen.com> Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> Signed-off-by: jooho lee <jlee@redhat.com> Signed-off-by: Jooho Lee <jlee@redhat.com> Signed-off-by: Jooho Lee <ljhiyh@gmail.com> Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> Signed-off-by: jordanyono <jordanyono@gmail.com> Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com> Signed-off-by: udai <udaij12@gmail.com> Signed-off-by: yxia216 <yxia216@bloomberg.net> Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com> Signed-off-by: Johnu George <johnugeorge109@gmail.com> Signed-off-by: Gavin Li <yli1109@bloomberg.net> Signed-off-by: Spolti <fspolti@redhat.com> Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com> Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Signed-off-by: Arjun Bhalla <arjunbhallasg@gmail.com> Signed-off-by: Snehomoy <snehomoy6@gmail.com> Signed-off-by: ops-jaeha <me@jaehai.com> Signed-off-by: Kursat Aktas <kursat.ce@gmail.com> Signed-off-by: oplushappy <ken910304@gmail.com> Signed-off-by: Sivanantham <90966311+sivanantha321@users.noreply.github.com> Signed-off-by: carlory <baofa.fan@daocloud.io> Signed-off-by: bentohset <toh.benjamin123@gmail.com> Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com> Signed-off-by: Kevin Mingtarja <69668484+kevinmingtarja@users.noreply.github.com> Co-authored-by: gfkeith <gregoryfkeith@gmail.com> Co-authored-by: Andrews Arokiam <87992092+andyi2it@users.noreply.github.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Jin Dong <greenmoon55@users.noreply.github.com> Co-authored-by: Sivanantham <90966311+sivanantha321@users.noreply.github.com> Co-authored-by: Jooho Lee <jlee@redhat.com> Co-authored-by: jordanyono <40174853+jyono@users.noreply.github.com> Co-authored-by: Curtis Maddalozzo <cmaddalozzo@users.noreply.github.com> Co-authored-by: Edgar Hernández <ehernand@redhat.com> Co-authored-by: udaij12 <32673964+udaij12@users.noreply.github.com> Co-authored-by: hustxiayang <yxia216@bloomberg.net> Co-authored-by: tjandy98 <3953059+tjandy98@users.noreply.github.com> Co-authored-by: Johnu George <johnugeorge109@gmail.com> Co-authored-by: Gavin Li <yueqi.li.usa@gmail.com> Co-authored-by: Filippe Spolti <fspolti@redhat.com> Co-authored-by: Arjun Bhalla <arjunbhallasg@gmail.com> Co-authored-by: Arjun Bhalla <abhalla31@bloomberg.net> Co-authored-by: Snehomoy.M <snehomoy6@gmail.com> Co-authored-by: 이재하 <me@jaehai.com> Co-authored-by: Kursat Aktas <kursat.ce@gmail.com> Co-authored-by: oplushappy <91440425+oplushappy@users.noreply.github.com> Co-authored-by: 杨朱 · Kiki <baofa.fan@daocloud.io> Co-authored-by: Benjamin Toh <115091471+bentohset@users.noreply.github.com>

* explicitly give a chat template Signed-off-by: yxia216 <yxia216@bloomberg.net> * fix dummy model issue, fix python version smaller than 3.10, and formatting Signed-off-by: yxia216 <yxia216@bloomberg.net> * fix vLLMModel Signed-off-by: yxia216 <yxia216@bloomberg.net> * change the interface of CreateChatCompletionRequest Signed-off-by: yxia216 <yxia216@bloomberg.net> * update dummy model's para Signed-off-by: yxia216 <yxia216@bloomberg.net> * consitent with OpenAIGPTTokenizer and OpenAIGPTModel Signed-off-by: yxia216 <yxia216@bloomberg.net> * give a chat template if there is no Signed-off-by: yxia216 <yxia216@bloomberg.net> * update the response and update the readme Signed-off-by: yxia216 <yxia216@bloomberg.net> * update the chat_template Signed-off-by: yxia216 <yxia216@bloomberg.net> * update data Signed-off-by: yxia216 <yxia216@bloomberg.net> * add test of chat temmplate for tokenizer Signed-off-by: yxia216 <yxia216@bloomberg.net> * jinja2 template format Signed-off-by: yxia216 <yxia216@bloomberg.net> * use a simpler chat template --------- Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang force-pushed the vLLM0.6.2 branch from 20322fd to 0299c07 Compare September 30, 2024 16:43

terrytangyuan reviewed Sep 30, 2024

View reviewed changes

hustxiayang changed the title ~~bump to vllm0.6.2~~ bump to vllm0.6.2; add explicit chat template for apply_chat_template method. Sep 30, 2024

spolti reviewed Oct 1, 2024

View reviewed changes

hustxiayang force-pushed the vLLM0.6.2 branch from 4e6642e to 796764f Compare October 5, 2024 17:49

hustxiayang changed the title ~~bump to vllm0.6.2; add explicit chat template for apply_chat_template method.~~ bump to vllm0.6.2; add explicit chat template in CreateChatCompletionRequest class Oct 5, 2024

hustxiayang force-pushed the vLLM0.6.2 branch from a2f97a7 to 4783136 Compare October 7, 2024 04:42

hustxiayang marked this pull request as draft October 7, 2024 16:48

hustxiayang force-pushed the vLLM0.6.2 branch 15 times, most recently from e21db9e to 64266a5 Compare October 9, 2024 04:11

hustxiayang changed the title ~~bump to vllm0.6.2; add explicit chat template in CreateChatCompletionRequest class~~ bump to vllm0.6.2; add explicit chat template in CreateChatCompletionRequest class;set clean_up_tokenization_spaces in tokenizer Oct 9, 2024

hustxiayang added 21 commits October 15, 2024 17:09

fix vLLMModel

3a7df66

Signed-off-by: yxia216 <yxia216@bloomberg.net>

format

39a6aa3

Signed-off-by: yxia216 <yxia216@bloomberg.net>

change the interface of CreateChatCompletionRequest

9f0998e

Signed-off-by: yxia216 <yxia216@bloomberg.net>

format, fix parameter

032a667

Signed-off-by: yxia216 <yxia216@bloomberg.net>

update dummy model's para

20472f4

Signed-off-by: yxia216 <yxia216@bloomberg.net>

consitent with OpenAIGPTTokenizer and OpenAIGPTModel

9781b8c

Signed-off-by: yxia216 <yxia216@bloomberg.net>

give a chat template if there is no

0abfaf3

Signed-off-by: yxia216 <yxia216@bloomberg.net>

test prompt

62205c7

Signed-off-by: yxia216 <yxia216@bloomberg.net>

lint

6f5e630

Signed-off-by: yxia216 <yxia216@bloomberg.net>

update the response and update the readme

5b21997

Signed-off-by: yxia216 <yxia216@bloomberg.net>

update the chat_template

6640cea

Signed-off-by: yxia216 <yxia216@bloomberg.net>

update data

d787a82

Signed-off-by: yxia216 <yxia216@bloomberg.net>

update json

79c4dd1

Signed-off-by: yxia216 <yxia216@bloomberg.net>

add test of chat temmplate for tokenizer

9f07905

Signed-off-by: yxia216 <yxia216@bloomberg.net>

typo

c013d32

Signed-off-by: yxia216 <yxia216@bloomberg.net>

template format

f9f836e

Signed-off-by: yxia216 <yxia216@bloomberg.net>

jinja2 template format

2ac1105

Signed-off-by: yxia216 <yxia216@bloomberg.net>

jinja2 template format

cf31e41

Signed-off-by: yxia216 <yxia216@bloomberg.net>

revert input data

fe34ce9

Signed-off-by: yxia216 <yxia216@bloomberg.net>

use a simpler chat template

aad523b

Signed-off-by: yxia216 <yxia216@bloomberg.net>

lint

60598c9

Signed-off-by: yxia216 <yxia216@bloomberg.net>

hustxiayang force-pushed the vLLM0.6.2 branch from 5d433f5 to 60598c9 Compare October 15, 2024 21:10

yuzisun approved these changes Oct 16, 2024

View reviewed changes

yuzisun merged commit 05dea67 into kserve:master Oct 16, 2024
60 checks passed

yuzisun changed the title ~~bump to vllm0.6.2; add explicit chat template in CreateChatCompletionRequest class; clean_up_tokenization_spaces were set as False by transformers~~ bump to vllm0.6.2 and add explicit chat template Oct 16, 2024

hustxiayang deleted the vLLM0.6.2 branch October 28, 2024 16:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bump to vllm0.6.2 and add explicit chat template #3964

bump to vllm0.6.2 and add explicit chat template #3964

hustxiayang commented Sep 27, 2024 •

edited by yuzisun

Loading

hustxiayang commented Sep 27, 2024

hustxiayang commented Sep 30, 2024

terrytangyuan left a comment

hustxiayang commented Sep 30, 2024

spolti left a comment

hustxiayang commented Oct 1, 2024

johnugeorge commented Oct 6, 2024 •

edited

Loading

hustxiayang commented Oct 7, 2024

yuzisun left a comment

yuzisun commented Oct 16, 2024

bump to vllm0.6.2 and add explicit chat template #3964

bump to vllm0.6.2 and add explicit chat template #3964

Conversation

hustxiayang commented Sep 27, 2024 • edited by yuzisun Loading

hustxiayang commented Sep 27, 2024

hustxiayang commented Sep 30, 2024

terrytangyuan left a comment

Choose a reason for hiding this comment

hustxiayang commented Sep 30, 2024

spolti left a comment

Choose a reason for hiding this comment

hustxiayang commented Oct 1, 2024

johnugeorge commented Oct 6, 2024 • edited Loading

hustxiayang commented Oct 7, 2024

yuzisun left a comment

Choose a reason for hiding this comment

yuzisun commented Oct 16, 2024

hustxiayang commented Sep 27, 2024 •

edited by yuzisun

Loading

johnugeorge commented Oct 6, 2024 •

edited

Loading