get kserve up to date #66
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kserve#3954) Signed-off-by: Gregory Keith <gregory.keith@maxkelsen.com>
) * initial commit for hugging face model download and load Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * bug fix on storage initializer Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * added hf_token and unittests Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * separate hf-storage-initializer image to reduce image size Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * review comment changes Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * snapshot download Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * use existing image for storage initializer Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * resolved merge conflicts Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * added hf storage uri validation Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * resolved merge conflicts Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> --------- Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com>
Signed-off-by: Dan Sun <dsun20@bloomberg.net>
* Consolidate into one commit Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix configmap format Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix configmap Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Log configmap read error Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * fix naming Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Update comments Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add enabled flag to configmap and avoid cluster resource check in isvc defaulter Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * move client into the local model block Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Fix lint Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net>
* Sync helm chart with kustomize Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update manifest generation script to sync helm charts Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Make kserve-addressable-resolver role optional Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Prepare for 0.14.0-rc1 release Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update release process Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Comment out crd sync script in make Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix helm template syntax Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> --------- Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
* add a new API for multi-node/multi-gpu Signed-off-by: jooho lee <jlee@redhat.com> * fix gitaction Signed-off-by: jooho lee <jlee@redhat.com> * fix merging conflict Signed-off-by: jooho lee <jlee@redhat.com> * fix gitaction fail Signed-off-by: jooho lee <jlee@redhat.com> * regenerate codegen/manifests Signed-off-by: jooho lee <jlee@redhat.com> * Apply suggestions from code review Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Jooho Lee <ljhiyh@gmail.com> * remove unnecessary comment Signed-off-by: jooho lee <jlee@redhat.com> * change the type of workerSpec in isvc to PodSpec Signed-off-by: jooho lee <jlee@redhat.com> * update controller-gen version Signed-off-by: jooho lee <jlee@redhat.com> * remove replicas from workerSpec Signed-off-by: jooho lee <jlee@redhat.com> * fix conflict merging Signed-off-by: jooho lee <jlee@redhat.com> * added size(replicas) for workerSpec again Signed-off-by: jooho lee <jlee@redhat.com> * add WorkerSpec to inferenceService Signed-off-by: jooho lee <jlee@redhat.com> * fix go linter Signed-off-by: jooho lee <jlee@redhat.com> --------- Signed-off-by: jooho lee <jlee@redhat.com> Signed-off-by: Jooho Lee <jlee@redhat.com> Signed-off-by: Jooho Lee <ljhiyh@gmail.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net>
…#3924) * fix openapigen.sh that can be executed from kserve dir Signed-off-by: jooho lee <jlee@redhat.com> * regenerate codegen/manifests Signed-off-by: jooho lee <jlee@redhat.com> * Update go.sum Signed-off-by: Dan Sun <dsun20@bloomberg.net> --------- Signed-off-by: jooho lee <jlee@redhat.com> Signed-off-by: Dan Sun <dsun20@bloomberg.net> Co-authored-by: Dan Sun <dsun20@bloomberg.net>
* Support python 3.12 Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update dependencies Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update deps to support 3.12 Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Remove python 3.8 support Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Remove skip for infer client test Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix port forward Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix sklearn pandas dep Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * skip pydantic v1 test for py 3.12 Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add setuptools dep for pmml Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix lgb Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Include setuptools for paddle Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Include setuptools for huggingface Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Rebase Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Rebase Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> --------- Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
* Bump version to 0.13.0-rc0 (kserve#3665) Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> Signed-off-by: jordanyono <jordanyono@gmail.com> * fixing docs Signed-off-by: jordanyono <jordanyono@gmail.com> * fix spelling mistake Signed-off-by: jordanyono <jordanyono@gmail.com> --------- Signed-off-by: Curtis Maddalozzo <cmaddalozzo@bloomberg.net> Signed-off-by: jordanyono <jordanyono@gmail.com> Co-authored-by: Curtis Maddalozzo <cmaddalozzo@users.noreply.github.com>
* Fix local testing Signed-off-by: Dan Sun <dsun20@bloomberg.net> * Fix codegen Signed-off-by: Dan Sun <dsun20@bloomberg.net> --------- Signed-off-by: Dan Sun <dsun20@bloomberg.net>
Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
* Add a flag for automount serviceaccount Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Set default to false Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Default to true Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix test error Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Update openapi generated.go Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix python lint Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Fix config loading Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>
…ainer (kserve#3985) * Do not set security context on the storage initializer from user container Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * Add securityContext to the default storage container in the helm chart Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>
This adds the model container as an init-container to mitigate a race condition that would happen if the model container is not present on the cluster-node. The race condition happens if the cluster is able to fetch and start the runtime container before the modelcar is fetched. This would lead to the runtime to terminate with error. By configuring the model container as an init-container the runtime won't start until the modelcar is fetched. Although there is still the risk of a race condition when the cluster schedules the runtime container first, the pod should stabilize after a few restarts of the runtime container and should either prevent a CrashLoopBackOff event on the pod, or the crash event would finish quickly. This improves compatibility with the runtimes which can now stay agnostic to the modelcar implementation, until better techniques (like native sidecars, and oci volume mounts) become mature. Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com>
* Initial commit for headers passing issue Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * modifying the e2e test for rebase conflict Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * bug fix on unittest Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * review changes Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * fix for test failure Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * bug fix on e2e test Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * overridding the entrypoint of custom model images Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * custom response header Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * fix for unittest failure Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * added custom response headers in post process Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * added predict time latency in example response header Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> * fix OOM --------- Signed-off-by: Andrews Arokiam <andrews.arokiam@ideas2it.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net>
* security update Signed-off-by: udai <udaij12@gmail.com> * adding sign off Signed-off-by: udai <udaij12@gmail.com> --------- Signed-off-by: udai <udaij12@gmail.com>
Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>
* temp commit Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> * python-release.sh Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com> --------- Signed-off-by: Jin Dong <greenmoon55@users.noreply.github.com>
* explicitly give a chat template Signed-off-by: yxia216 <yxia216@bloomberg.net> * fix dummy model issue, fix python version smaller than 3.10, and formatting Signed-off-by: yxia216 <yxia216@bloomberg.net> * fix vLLMModel Signed-off-by: yxia216 <yxia216@bloomberg.net> * change the interface of CreateChatCompletionRequest Signed-off-by: yxia216 <yxia216@bloomberg.net> * update dummy model's para Signed-off-by: yxia216 <yxia216@bloomberg.net> * consitent with OpenAIGPTTokenizer and OpenAIGPTModel Signed-off-by: yxia216 <yxia216@bloomberg.net> * give a chat template if there is no Signed-off-by: yxia216 <yxia216@bloomberg.net> * update the response and update the readme Signed-off-by: yxia216 <yxia216@bloomberg.net> * update the chat_template Signed-off-by: yxia216 <yxia216@bloomberg.net> * update data Signed-off-by: yxia216 <yxia216@bloomberg.net> * add test of chat temmplate for tokenizer Signed-off-by: yxia216 <yxia216@bloomberg.net> * jinja2 template format Signed-off-by: yxia216 <yxia216@bloomberg.net> * use a simpler chat template --------- Signed-off-by: yxia216 <yxia216@bloomberg.net>
Signed-off-by: yxia216 <yxia216@bloomberg.net>
* Add hf transfer Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com> * Add hf transfer env Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com> --------- Signed-off-by: tjandy98 <3953059+tjandy98@users.noreply.github.com>
Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Signed-off-by: Johnu George <johnugeorge109@gmail.com>
* init CR Signed-off-by: Gavin Li <yli1109@bloomberg.net> * make generate Signed-off-by: Gavin Li <yli1109@bloomberg.net> * make manifests Signed-off-by: Gavin Li <yli1109@bloomberg.net> * black format Signed-off-by: Gavin Li <yli1109@bloomberg.net> * fix generated python code Signed-off-by: Gavin Li <yli1109@bloomberg.net> * feedback Signed-off-by: Gavin Li <yli1109@bloomberg.net> * more feedback Signed-off-by: Gavin Li <yli1109@bloomberg.net> * black format Signed-off-by: Gavin Li <yli1109@bloomberg.net> * make manifests Signed-off-by: Gavin Li <yli1109@bloomberg.net> --------- Signed-off-by: Gavin Li <yli1109@bloomberg.net>
Reduce E2Es dependency on CI environment Some code of the E2Es assume the environment is GitHub, because it is referring to GitHub-specific variables. This PR focuses on the `kserve/custom-model-grpc` container image, so that no Python code of the E2Es using this image is referencing the `github_sha` variable. Also, a small improvement on the `get_isvc_endpoint` utility function is done to use the schema in the endpoint specified in the status of the InferenceService, rather than hard-coding to plain-text HTTP. This adds compatibility for CI environments where KServe ConfigMap has been configured with `urlScheme: https` for the Ingress. Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com>
allow gcs to download single file fixes kserve#4013 Signed-off-by: Spolti <fspolti@redhat.com>
Signed-off-by: yxia216 <yxia216@bloomberg.net>
…kserve#4020) * Set default for SamplingParams.max_tokens in OpenAI requests if unset Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com> * Fix lint Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com> * Fix formatting Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com> --------- Signed-off-by: Kevin Mingtarja <kevinm@hypermode.com>
* Add tools to chat template Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Linting Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> add test Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Fix linting manually Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> * Fix linting Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> --------- Signed-off-by: Arjun Bhalla <abhalla31@bloomberg.net> Signed-off-by: Arjun Bhalla <arjunbhallasg@gmail.com> Co-authored-by: Arjun Bhalla <abhalla31@bloomberg.net>
…rve#4012) * Fix readiness probe logic and update test scenarios for HTTPGet, TCPSocket, and Exec handling Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Update: Refactor logic for readiness probe handling Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Apply gofmt formatting to agent_injector.go Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Added logger to replace fmt.Printf for better consistency and observability Signed-off-by: Snehomoy <snehomoy6@gmail.com> * Formatted file using goimports with -local Signed-off-by: Snehomoy <snehomoy6@gmail.com> --------- Signed-off-by: Snehomoy <snehomoy6@gmail.com>
) (kserve#4018) * Feat: Fix memory issue by replacing io.ReadAll with io.Copy (kserve#4017) Previously, io.ReadAll was causing out-of-memory problems when downloading large files from GCS. This change replaces io.ReadAll() with io.Copy() to stream data and prevent excessive memory usage. Signed-off-by: ops-jaeha <me@jaehai.com> * Feat: Fix add newline at end of file to satisfy golang lint Signed-off-by: ops-jaeha <me@jaehai.com> * Feat: Refact log Info for golang lint (kserve#4017) Signed-off-by: ops-jaeha <me@jaehai.com> --------- Signed-off-by: ops-jaeha <me@jaehai.com>
chore: Fix CVE-2024-26130 - NULL Pointer Dereference - Upgrade cryptography to version 42.0.4 or higher. Update Python version to match KServe 0.14.0 Update tensorflow, tensorflow-io-gcs-filesystem and dill libraries Signed-off-by: Spolti <fspolti@redhat.com>
Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
Signed-off-by: Kursat Aktas <kursat.ce@gmail.com>
…rve#4024) * Fix huggingface srever not work with return_probabilities Signed-off-by: oplushappy <ken910304@gmail.com> * Fix pytest huggingface server assertion error Signed-off-by: oplushappy <ken910304@gmail.com> * Fix the lint error and Add approx for assertion Signed-off-by: oplushappy <ken910304@gmail.com> * Parse string output to dictionary for accurate assertion Signed-off-by: oplushappy <ken910304@gmail.com> * Fix linting error Signed-off-by: oplushappy <ken910304@gmail.com> --------- Signed-off-by: oplushappy <ken910304@gmail.com>
* Add deeper readiness and liveness check for transformer Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add unit tests Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * put the feature behind flag Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update tests Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * resolve comments Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Make use of inference client Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add e2e test Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Make inference client singleton and lazy initialize Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Raise 503 If server is not ready / live Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Add test for custom transformer with rest protocol Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Fix CI running out of space Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Increase memory limit Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Check for model ready Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Webhook debug Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Address reviews Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Check for retry count in grpc client Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> * Update python/kserve/kserve/model_server.py Co-authored-by: Dan Sun <dsun20@bloomberg.net> Signed-off-by: Sivanantham <90966311+sivanantha321@users.noreply.github.com> --------- Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com> Signed-off-by: Sivanantham <90966311+sivanantha321@users.noreply.github.com> Co-authored-by: Dan Sun <dsun20@bloomberg.net>
…#4006) chore: Fix CVE-2024-47874 Signed-off-by: Spolti <fspolti@redhat.com>
remove duplicated import Signed-off-by: carlory <baofa.fan@daocloud.io>
Signed-off-by: Sivanantham Chinnaiyan <sivanantham.chinnaiyan@ideas2it.com>
* add storageaccesskey to azure env builder Signed-off-by: bentohset <toh.benjamin123@gmail.com> * update integration and unit test for azure storage access key Signed-off-by: bentohset <toh.benjamin123@gmail.com> * fix formatting Signed-off-by: bentohset <toh.benjamin123@gmail.com> --------- Signed-off-by: bentohset <toh.benjamin123@gmail.com>
* support single digit azure zone id Signed-off-by: bentohset <toh.benjamin123@gmail.com> * add single digit azure dns zone id tests Signed-off-by: bentohset <toh.benjamin123@gmail.com> * fix formatting Signed-off-by: bentohset <toh.benjamin123@gmail.com> --------- Signed-off-by: bentohset <toh.benjamin123@gmail.com>
Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com>
Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com>
Signed-off-by: Kevin Mingtarja <kevin.mingtarja@gmail.com>
* improve dockerfile, makefile, readme * support custom classification labels, refactor postprocess * support text embedding task * improve support for token classification (named entity recognition)
Signed-off-by: Kevin Mingtarja <69668484+kevinmingtarja@users.noreply.github.com>
mangalaman93
approved these changes
Nov 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.