-
Notifications
You must be signed in to change notification settings - Fork 320
[Bugfix][Router]: loop through model_names #681
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bugfix][Router]: loop through model_names #681
Conversation
89aa6f9 to
4c9eeda
Compare
| logger.debug("==== Total endpoints ====") | ||
| logger.debug(endpoints) | ||
| logger.debug("==== Total endpoints ====") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This MR also removes a lot of excessive logging, while being more transparent about errors
f3727d2 to
4da5ad3
Compare
633bcd0 to
9472fd0
Compare
Signed-off-by: Max Wittig <max.wittig@siemens.com>
9472fd0 to
8e02e5c
Compare
Signed-off-by: Max Wittig <max.wittig@siemens.com>
1cdb877 to
136bd7c
Compare
Signed-off-by: Max Wittig <max.wittig@siemens.com>
136bd7c to
196ea0f
Compare
|
@YuhanLiu11 We have this running in our dev environment. Let me know what you think about this bugfix. |
|
@max-wittig hey I understand that we need to modify route_general_transcriptions to fix this bug, but why do we need to modify service_discovery.py for this bug fix? |
|
@YuhanLiu11 Because we need to use It modifies |
|
Split up into the first part: #694 |
|
Hey @max-wittig yes, splitting it to two PRs sounds good! Thanks |
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
As model_label is already used for things related to the LMCache. It makes more sense to use the type of the model here. Replaces vllm-project#681 Signed-off-by: Max Wittig <max.wittig@siemens.com>
Fixes this crash.
model_namedoes not exist:Tested on our dev system:
-swhen doinggit commit[Bugfix],[Feat], and[CI].Detailed Checklist (Click to Expand)
Thank you for your contribution to production-stack! Before submitting the pull request, please ensure the PR meets the following criteria. This helps us maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:
[Bugfix]for bug fixes.[CI/Build]for build or continuous integration improvements.[Doc]for documentation fixes and improvements.[Feat]for new features in the cluster (e.g., autoscaling, disaggregated prefill, etc.).[Router]for changes to thevllm_router(e.g., routing algorithm, router observability, etc.).[Misc]for PRs that do not fit the above categories. Please use this sparingly.Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:
pre-committo format your code. SeeREADME.mdfor installation.DCO and Signed-off-by
When contributing changes to this project, you must agree to the DCO. Commits must include a
Signed-off-by:header which certifies agreement with the terms of the DCO.Using
-swithgit commitwill automatically add this header.What to Expect for the Reviews
We aim to address all PRs in a timely manner. If no one reviews your PR within 5 days, please @-mention one of YuhanLiu11
, Shaoting-Feng or ApostaC.