LLM Instance Gateway Proposal #8029
Conversation
k8s-ci-robot
added
the
cncf-cla: no
|
Welcome @kfswain! |
k8s-ci-robot
added
the
needs-ok-to-test
|
Hi @kfswain. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/L
k8s-ci-robot
added
the
wg/serving
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kfswain The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
|
|
||
| waiting for a new | ||
| model server to start. | ||
| - Efficient accelerator sharing - Use cases can use less than an accelerator |
There was a problem hiding this comment.
what's the use case of "use less than an accelerator"?
There was a problem hiding this comment.
A model use-case or LoRA adapter that doesn't utilize the full batch size of an accelerator. This is meant to capture the value prop around why sharing accelerators is valuable (a single use-case will not always get the maximum value out of an accelerator at all times).
There was a problem hiding this comment.
got you. At first glance, I mistook it for some other GPU sharing techniques.
| with this LLM Instance Gateway, the model server would need to implement this | ||
| protocol. | ||
|
|
||
| Key requirements would roughly look like: |
There was a problem hiding this comment.
em. I am thinking such protocol adaptation takes time to land into the inference server. Should we consider the routing only instead of lora management in the engine?
There was a problem hiding this comment.
/cc @varungup90 this is the protocol from feature list
There was a problem hiding this comment.
Agree this is a big rock, but I would like to make native support a goal for vllm. We should cover in the architecture proposal.
|
@Jeffwan: GitHub didn't allow me to request PR reviews from the following users: protocol, from, feature, list, varungup90, this, is, the. Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Having an efficient and reliable LLM Gateway is essential. It serves as a bridge between inference engine and LLM management in scale. Bytedance is passionate about supporting innovative ideas like the LLM Gateway and are excited to participate in this project. By doing so, we aim to contribute to the development of routing algorithm, envoy extension, lora integration with engine that can bring significant benefits to LLM users on kubernetes. |
k8s-ci-robot
added
cncf-cla: yes
|
Hey all! I'm closing this PR as we now have: kubernetes-sigs/wg-serving#12. Thanks! |
8 participants
Creating a proposal for the LLM Instance Gateway, as discussed in the wg-serving Wednesday meetings