Skip to content

Improve configs - ModelConfig #17130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 53 commits into from
Apr 30, 2025
Merged

Improve configs - ModelConfig #17130

merged 53 commits into from
Apr 30, 2025

Conversation

@hmellor
Copy link
Member

@hmellor hmellor commented Apr 24, 2025 •
edited by github-actions bot
Loading

Follows #16332, #16422, #16533, #16603, #16789, #16835, #16971, #16980

Notable changes:

  • Type hints on QuantizationConfig and LLM classes have been made more specific, using a Literal of possible options instead of str
  • ModelConfig is no longer used as a key to a dictionary in the multimodal registry (keys are usually simple and hashable, a large dataclass is neither of those things)

Changes derived from this PR (this PR should only be merged after the PRs below have merged):

@hmellor
Initial commit
9ed887a 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@github-actions
Copy link

github-actions bot commented Apr 24, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@mergify mergify bot added the frontend label Apr 24, 2025
hmellor added 22 commits April 24, 2025 22:24
@hmellor
Second commit
4550a8c 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Merge branch 'main' into improve-model-config
a75f74e 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Fix return type of optional_type
6304a8d 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Use Literal types instead of str in LLM
947b226 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Use Literal instead of str for override_quantization_method
1fb17d9 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add cast when assigning to Literal in some cases
f00d97d 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Better handling of tokenizer falling back to model
4fc786c 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add missing pooler_config field and better handle its override
eadc39f 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Max model len will never be None by the time it's used
039d89d 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
max_model_len default is already None
41bc717 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
quantization in SpeculativeConfig can also be Literally typed
378809c 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add nargs to tuple arguments
bf6e36a 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
field_type -> type_hint, better extraction from optional, add list type
9bcac01 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Fix type reassignment
3e061ce 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Make naming of compressed_tensors consistent
8a6e4df 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Stronger typing of QuantizationConfigs
1fe5a04 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Don't assign tuple to list
e6b7a51 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add QuantizationMethods to __all__ because it's used externally
c342876 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Handle ordering of overrides
2f7f6de 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Use enum values for defaults
7a1b442 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Fix config_format typing
1f3e9ba 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Fix hf token arg
96ebdec 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor hmellor marked this pull request as ready for review April 25, 2025 17:45
@mergify mergify bot added the tpu Related to Google TPUs label Apr 25, 2025
hmellor added 16 commits April 29, 2025 11:17
@hmellor
Fix union of literals case
beab5ae 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Fix arg_utils test
5a7e339 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Enforce that Literals are merged with Literal not Union
6df5dbb 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add tests for config decorator
08d5e20 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Create new helper function to handle sequences of literals
92300b6 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add test for literal to kwarg
88c1479 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add test cases for list[Literal] and Literal[Literal, Literal]
4037013 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Fix pre-commit
0ecf76e 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Respond to comment
8beb8df 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Merge branch 'config-literal-handling' into improve-model-config
4510fa1 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Merge branch 'main' into improve-model-config
cb08bdc
@hmellor
Remove change that was handled in another PR
1232ded 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Add change from other PR
0055700 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Merge branch 'main' into improve-model-config
7d6108a 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Remove now unneeded test case
536f05c 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@hmellor
Remove test which is just testing json.loads
1282937 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
@DarkLight1337 DarkLight1337 merged commit 13698db into vllm-project:main Apr 30, 2025
57 checks passed
@hmellor hmellor deleted the improve-model-config branch April 30, 2025 06:58
@hmellor hmellor mentioned this pull request Apr 30, 2025
Merged
radeksm pushed a commit to radeksm/vllm that referenced this pull request May 2, 2025
@hmellor @radeksm
Improve configs - ModelConfig (vllm-project#17130)
355d0a8 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@hmellor
Improve configs - ModelConfig (vllm-project#17130)
f6417b0 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
lk-chen added a commit to lk-chen/ray that referenced this pull request May 23, 2025
@lk-chen
fix test due to vllm-project/vllm#17130
cfee92a 
Signed-off-by: Linkun Chen <github@lkchen.net>
zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025
@hmellor
Improve configs - ModelConfig (vllm-project#17130)
39199a0 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96

Assignees

No one assigned

Labels

frontend multi-modality Related to multi-modality (#4194) ready ONLY add when PR is ready to merge/full CI is needed tpu Related to Google TPUs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hmellor @DarkLight1337